Quote from: BadBitcoin (James Sutton) on February 04, 2014, 01:50:06 AM

bitfreak: I'm nearly done my foundations, I've written an RNN from scratch (nearly got it to work, just going to test for bugs tomorrow). Strongly contemplating making this an open source project but I'm on the fence being that I'm literally sleeping on my dads couch until he throws me out. Looks really good though, can't wait to see how it looks finished.

I would be interested in seeing your source code but don't make it open source if you don't want to. What I designed is also a type of recurrent neural network, although I think it's a bit different from most other types of RNN's. I'll give you an example of what one of my nets would look like. First of all I want to show you what the DNA code for a net looks like, here is an example DNA string for a very simple net:

I:0/I:0|0:1-1/0:1;2-3/0:1;2-1/0:1-2/0:1;2-3/C:4;5/B:1|1:3;4-2/1:1;4-1/1:1;5-1/1:1;2-3/1:3;4-3/C:5;5/B:1|2:1;2;3;4;5-2/B:1#4,23,13

I designed it so that random DNA strings can easily be generated and stored in a simple text file and so that a net object can easily be constructed from the DNA. First I will quickly explain the format of the DNA string and then I'll post a picture which shows the actual structure of the net associated with the DNA string above. The weights are stored in a separate file from the DNA, but I wont explain the format of the weight text because it's fairly simple to understand and not really important to this explanation.

Everything after the # character is "meta data" for the net. So we have the numbers 4, 23 and 13 separated by commas after the # character. What that means is that the net has 4 layers, 23 connections (not including bias connections and context connections) and 13 neurons (not including bias neurons and context neurons). Each layer in the DNA string is separated by the | character, so if we remove the meta data and then split the string into substrings using | as the separator we get the following strings of text:

I:0/I:0
0:1-1/0:1;2-3/0:1;2-1/0:1-2/0:1;2-3/C:4;5/B:1
1:3;4-2/1:1;4-1/1:1;5-1/1:1;2-3/1:3;4-3/C:5;5/B:1
2:1;2;3;4;5-2/B:1

So now it becomes clear that we have 4 layers in this particular example. Then as you may have already guessed, each neuron is separated by the / character. The two neurons in the first layer are obviously the input neurons, I:0 means that it is an input neuron with a default value of 0 (kind of pointless to have a default value but I wanted consistency in the format). The second and third layers both have 5 neurons each as well as one context neuron and one bias neuron. B:1 ones obviously means a bias neuron with an output value of 1.

The context neurons are a little bit more complicated, if we take the context neuron in the second layer it reads "C:4;5" which means it's a context neuron with a link to the 4th neuron in the the layer in which it exists and it will remember and feed back the output of that neuron for 5 iterations before it accepts a new output from the 4th neuron. In other words it will hold the same value for 5 iterations and then it will accept a new value from the 4th neuron in the 2nd layer and it will remember that value for 5 iterations and feed it back to the 4th neuron. We could have more than 1 context neuron on each layer but I've avoided that for this example.

Now if we look at the 5th neuron in the 2nd layer it has a value of "0:1;2-3". The 0 means that it has connections to neurons in layer 0 (the input layer) and the 1;2 part means that it connects to the 1st and 2nd neurons in layer 0 (so it has a connection to both the input neurons). So it's listing where connections come from, not where they go. The 3 after the - character refers to the activation function. Every neuron in this net can be assigned one of three different types of activation functions, meaning not all neurons need to have the same activation function. If a value of 0 is placed after the - character it means the neuron doesn't use an activation function at all.

It is even possible in my system to have a single neuron link to neurons which are separated by multiple layers. For example if one of the neurons in the 3rd layer (layer 2 if we start from 0) had a value of "0:1;2,1:1;3-3" then it would connect to neurons in the first two layers (layer 0 and layer 1) because it can be separated into "0:1;2" and "1:1;3" using the comma as the separator (keeping in mind I removed the -3 from the end because it's the activation function and not part of the connections). But I haven't included multi-layer connections in this example because it would make it a bit too messy.

I also left out the bias neurons and their connections from the following diagram because it would just make it unnecessarily messy and overly complex. But you can easily imagine the bias neurons feeding a value of 1 into all of the neurons (apart from the input neurons). The context neurons and their corresponding connections are shown in the lighter gray color. The numbers inside the context neurons represent their memory length in iterations. The numbers inside the rest of the neurons represent the type of activation function. The numbers inside the input neurons are 0 because they have no activation function.

Hopefully you understood my above explanation and you will be able to see how the DNA string example I gave above can be transformed into the following neural network: