A taste of theory
The underpinnings of an ANN is a huge collection of elements called neurons, typically arranged in bundles that are heavily interconnected. Briefly described, a neuron is a cell that is characterized by having multiple inputs and a single output. The output of the cell is a direct function of the inputs to it, each of which is getting different ‘attention’ in the overall contribution to the output. This level of ‘attention’ is usually referred to as weight. Additionally, the output may carry some thresholding effect that results in generating a response only if the neuron has crossed the threshold also known as ‘fired’. The relevant inputs of neurons down the line that are connected to a firing neuron, will get ‘excited’ and the process will carry throughout the network to reach an eventual output.
Figure 1: The neuron biological inspiration (left) and its artificial, conceptual equivalent (right). The dendrites serve as the inputs; the axon is the output and the aggregation takes place within the cell.
While defining the equivalent model, the most common approach is weighted sum with a non-linearity applied to the output. This approach is very useful in capturing the essence of a concept in a simplistic and meaningful manner. However, in attempts to capture finer aspects of the biological behavior more complex models are sought. These reflect other properties that may result in a more complete description of the neuron and for practical reasons may offer implementation alternatives that overcome some performance barriers inherent to the basic representation.
Options to model the neuron behavior involve time-domain, frequency-domain and amplitude-domain representations. These options can be easily expressed in closed mathematical form as described below.
The straightforward discrete model representing the neuron as a weighted sum of the inputs (figure 2a); A pulsed version where pulse-trains represent activity and their temporal rate determines level of excitation - this is the one that represents closest representation of nerve cell activity in the human body (figure 2b) and a continuous representation.
Figure 2: Mathematical representation of a) discrete (b) pulsed and (c) continuous models.