Very well written, and I applaud the effort. But personally I don't care for the "magical" aura that writers tend to give ANNs - to me, they are simply (non-linear) function approximators that have a nice fitting algorithm. They work well for some problems and poorly for others. Also, beware of over-fitting - ANNs tend to be parameter-heavy, although there are approaches to prune the connections.
If you liked his first chapter, consider supporting his IndieGogo campaign for the whole book (<a href="http://www.indiegogo.com/projects/neural-networks-and-deep-learning-book-project/" rel="nofollow">http://www.indiegogo.com/projects/neural-networks-and-deep-l...</a>).
<i>"The adder example demonstrates how a network of perceptrons can be used to simulate a circuit containing many NAND gates. And because NAND gates are universal for computation, it follows that perceptrons are also universal for computation."</i><p>I think this comment from the article needs caveats. Of course, a neural network would not qualify as Turing Complete just because it's finite. Keep in mind also that neural network, lacking anything like counters, tape, or recursion, couldn't approximate a Turing in the way that a finite Von Neuman architecture machine does. (A NN can represent any given function over a domain if it get large enough, kind of the universality of a finite automaton).<p>I know this a reference to this generation of NN having overcome an earlier problem of <i>not</i> being able to represent a NAND gate but still, it's worthing keeping mind that an ordinary computer can simulate an NN with just a program but this doesn't work vice-versa, so that NN's in that sense are far from universal.
This is a cool exercise! After completing it, I wanted to find out exactly what each NN hidden node represented. I trained a tiny (10 hidden node) NN on an OCR dataset and created a visualization here: <a href="https://rawgithub.com/tashmore/nn-visualizer/master/nn_visualizer.html" rel="nofollow">https://rawgithub.com/tashmore/nn-visualizer/master/nn_visua...</a> .<p>Can anyone figure out what each hidden node represents?<p>You can also select a node and press "A" (Gradient Ascent). This will change the input in a way that increases the selected node's value. By selecting an output node and mashing "A", you can run the NN in reverse, causing it to "hallucinate" a digit.
Ahh, the MNIST database of handwritten digits. I never took an ML course and I was only able to achieve 87% recognition rate 6 years ago for a university software engineering project. I read others achieving 99.9% recognition rates with their ANNs so I wasn't happy with my result. I tried to self-study about ANNs but I found most material to be either too simple or too complicated. I finally found some articles about ANNs (<a href="http://visualstudiomagazine.com/Articles/List/Neural-Network-Lab.aspx" rel="nofollow">http://visualstudiomagazine.com/Articles/List/Neural-Network...</a>) with code samples in C#, so I'll finally be looking into rewriting my old code to get a better result.
That reminds me of a video I saw about something called restricted boltzmann machines:<p><a href="http://www.youtube.com/watch?v=AyzOUbkUf3M&t=24m0s" rel="nofollow">http://www.youtube.com/watch?v=AyzOUbkUf3M&t=24m0s</a>
I'm not sure if I am making a mistake, but I couldn't use the command listed in order to clone the repository. I'm using windows, with git installed, and I received the error: "Permission Denied: publickey"<p>I was able to get everything by looking you up on github and using the url of the repository.<p>Edit: Also, you might mention the repository earlier, because it's rather large and I've had to break from the book while it downloads.
I did exactly this for a school project about a year ago:<p><a href="https://github.com/bcuccioli/neural-ocr" rel="nofollow">https://github.com/bcuccioli/neural-ocr</a><p>There's a paper in there that explains the design of the system and my results, which weren't great, probably due to the small size of training data.