[Back to Home]
| |
Sam Hill: Kohonen Neural Networks - Example Network [Back to Home] |
| ||
|
Sam Hill: Kohonen Neural Network Simulation The applet below is a demonstration of how the Kohonen Neural Network operates. The Kohonen Neural Network is a developemnt of the "percepton" network developed by Rosenblatt in 1962.Both networks make use of artificial neurons which are electrical simulations of biological ones - the core component of a brain. These artificial neurons are mathematical abstractions and aim to emulate the functionality of the neurons through computational code, this artificial abstraction is often termed neuro-computing, this term is used since neuro-computing is fundamentally different to digital computing. Neurons are what makes a brain a much more powerful processor than a computer, this is because by careful organisation of the neurons, the brain can operate much more efficiently than the most powerful of today’s computers. A neuron on its own is an incredibly simple object. It is made up of an input, and operator and an output in its basic form. This program aims to generate an Artificial Neural Network (ANN) that is capable of recognising given input sequences. These input sequences consist of sequences of letters making up familiar words an example input sequence could be h-e-l-l-o where the actual word is hello, made up from the letters: h, e, l, l and o. The neural network should then recognise this input sequence, if the sequence was contained in the training data for the network and produce a response to that input such as to the input Hello the network could respond with Hello to you. The neural network should, if properly trained be flexible about noise in the input sequence so that Hexxo would also be recognised as Hello. This program aims to generate an Artificial Neural Network (ANN) that is capable of recognising given input sequences. These input sequences consist of sequences of letters making up familiar words an example input sequence could be h-e-l-l-o where the actual word is hello, made up from the letters: h, e, l, l and o. The neural network should then recognise this input sequence, if the sequence was contained in the training data for the network and produce a response to that input such as to the input Hello the network could respond with Hello to you. The neural network should, if properly trained be flexible about noise in the input sequence so that Hexxo would also be recognised as Hello. This demonstration example of the Kohonen Neural Network has been developed using the Java programming language. How it all works... A neuron on its own is an incredibly simple object. It is made up of an input, and operator and an output in its basic form. This operator is a weighting system that given significant input, the neuron will fire – or give an output. A biological neuron is made up of dendrites (inputs), a cell body (the operator) and axons (outputs) and each neuron tends to be connected to 10,000s of others… An artificial neuron is made up of synapses (the equivalent of the dendrites and axons) and an adder (equivalent to the cell body).  Unlike biological neurons, artificial neurons tend to only be connected to a few other neurons, and also have weighing factors on the synapses to enable the neuron to operate correctly. These artificial neurons are what makes neuro-computing an artificial abstraction of the biological world – and are attempts at using mathematics to simulate and mimic the biological world. Suprisingly Artificial Neurons and actually operate a lot faster than their biological counterparts through the use of existing logical gates. The Kohonen Network model for Neural Networks is multiple lattice like layers of neurons, all connected to the same input to the system. With all inputs going into every neuron (this is a very basic single layer over-view diagram).  Weightings are then applied to every input line between a neuron and the input pattern - so in the diagram above, the weights on the input lines to the neuron might be 4, 5, 8, 1, 4, 3,7, 2, 6 and 9. Every neuron has different weights on the input lines between the input sequence and the neuron itself. It is these weights that are the key to the operation of a Kohonen network. When a network has been fully trained (training is explained subsequently), for any given input sequence to the network all the Neurons calculate their value, which is produced by this formula: Where the output to the Neuron is the sum of all its inputs, minus the weight on that input line, squared. The closer to the output from the Neuron is to 0, then the better the match for the input pattern.
To train the network, the input sequences required to be known by the network are passed to the Kohonen map many, many times - each time though, the weights on the winning neuron and those surrounding the neuron are adjusted by the formula: This causes the weights on the neuron to over time adjust and become close to, if not the same as the input sequence. The Gain term is a numerical value, which decreases over time (as in training iterations) which reduces the steps of convergence towards the answer - as in, for the first few cycles the weights would change dramatically for the neurons, after a few cycles the weight changes become smaller so the change over the network is reduced. Also the sphere of influence of the winning Neuron over its surrounding neighbours is also reduced. This neighbourhood of neurons allows noisy patters to be recognised by the system, as although it will not match up exactly with the winning neuron, it might match up with one of its surrounding neighbours of neurons. This also causes input sequences with similar characteristics to each other to begin to collect together in zones on the network. If the example above, is then expanded to a much grander scale say a lattice of neurons 10 by 10. Then neighbourhoods begin to form around the neurons due to the way training works. This neighbourhood of neurons all have similar outputs to the winning neuron although not as close to 0 as the winning neuron. If a graphical end is then placed over this network, the neurons can be plotted based on their output values - by assigning a scaled colour to these outputs, the neighbourhoods can be seen: 
Here, the winning neuron and those with similar outputs have a white or near white colour. By then applying Memory techniques and associated memory, the network can be completed. This memory is based on the neighbourhood surrounding the winning neuron - the smaller the overlap between zones the easier it is to divide the network up into sections. Each section dedicated to a particular input sequence. The way this network is used in the context of Sequence Recognition, Sequence Reproduction and Sequence Association is by firstly training the network to recognise a series of input sequences. Once trained (training is determined to be completed when the changes between the winning neurons for all the input sequences are small) the neighbourhoods for each sequence can be established which is then mapped into memory locations for that sequence. Then to use, a sequence is applied to the network, this then replied with a winning neuron, which in turn is looked up using the memory for the system from which a reply is created - this can be either a 1/0 for Sequence Recognition - a 1 if the neuron matches up to a valid memory location, for Sequence Reproduction, the memory would output the correct input sequence - such as Hello for Hello or Heffo. And, for Sequence Association, the memory would reply with Goodbye for Hello as the word hello has been associated with goodbye. Through a combination of different learning techniques, the neural network can be taught to function as an an artificial brain where it learns itself the correct responses for new input sequences. Supervised learning is where the neural network is taught using predetermined ranges of inputs and outputs. If the neural network then the weights are adjusted each iteration to teach the neuron to produce the right results for given inputs from a series of predefined inputs and outputs. This is based on the weights being given an initial value (a trial) which is then adjusted to produce the correct results. Unsupervised learning is where the network does not know the right answer for any given problem – or what its weights should be. Through subsequent further iterations trends can be discovered and these can be used to teach the network the right answers to the problems. Reinforcement Learning is the method that relies on trial and error. The weights between neurons that fire together more frequently are strengthened, and the weights on the neurons less used connections are reduced. This allows the dominant connections to take over and squash the potential firing of incorrect results from the system – through return feedback from one node into the others. This demonstration applet only makes use of Supervised learning and therefore can be deemed to not have any degree of Articifial Intelligence. How to use the Example Applet presented here: To test the network, training sets were taught to the network and the results compared with the desired outputs. Basically you teach the network that for a given input (which can be a fuzzy) the network needs to respond with correctly associated output for the sequence. For the Sequence Association, which is the most complex output sequence from a neural network, the following was given as inputs to the neural network: Goodbye -> See you later. Christmas -> jingle Bells. Millennium -> Big party! After the training is complete, the applet displays: (Training took roughly 5000 cycles):  Providing inputs to the network then tests this: Entering Hello produces as an output How are you?.  Entering Goodbye produces See you later.  Then testing noisy inputs to the system: Henno produces How are you?  As can be seen from these examples, the Neural Network could be said to operate correctly for the test data. This test was using a 25 x 25 network with an initial neighbourhood of 10. |
