"What does a neural network actually do?"<p>This is a fundamental Question:
Can we really say and predict what a neural network does?<p>Contrary to an engineered / constructed algorithm a neural network is 'trained'.<p>Whenever we will present a 'known' input pattern it will reposnd with a 'learned' response.<p>This however introduces interesting problems: How can we _debug_ a neural network? How can we debug a correlation? Sure we can tune its paramters, we can train it some more to again show the desired response. But reaching this point we just abandoned knowing how the intrinsic algorithm works in favor to just focussing on the result.<p>Okay - now if we follow this argument - this would lead to: If we simulate the whole brain by simulating the neural network, we won't gain any knowlege about the intrinsic workings of the brain. We won't find any enlightenment about the innermost algorithm _represented_ by the neural network we call our brain.
I've found that, in practice, traditional neural networks tend to be prone to overfitting and are finicky about their parameters (in particular the topology and number of nodes you choose).<p>I use the word "traditional" to describe the NN architecture discussed in the article. Recent NN research has been promising [1], but this article strictly discusses traditional NN's. I don't really have much experience with the newer NN algorithms, so I'm not sure to what extent they suffer from the same problems as traditional NN's.<p>[1] <a href="http://en.wikipedia.org/wiki/Neural_network#Recent_improvements" rel="nofollow">http://en.wikipedia.org/wiki/Neural_network#Recent_improveme...</a>
Another limit they don't address is that the training normally used is purely local— just a gradient descent. So even when the network can model your function well, there is no guarantee that it will find the solution.<p>For me ANN's always seem to get stuck on not very helpful local minima— they're not one of the first tools in my bags of tricks by far.<p>Often I associate them as being the sort of thing that someone who doesn't really know what they're talking about talks about. (Esp. if its clear that in their minds NN have magical powers. :) maybe they'll also mention something about "genetic algorithms")
A less mathy explanation with some real examples: <a href="http://neuralnetworksanddeeplearning.com/chap1.html" rel="nofollow">http://neuralnetworksanddeeplearning.com/chap1.html</a><p>Coding a digit recognizer using a neural network is an extremely rewarding exercise and there's a lot of help on the web to get you started.
This is a pretty good article, but I'm seeing a lot of confusion in this thread because the article is maybe one step ahead of the basic intuition needed to understand why ANNs are not magical and are not artificial intelligence (at least not feed-forward networks).<p>Perhaps a simpler way to look at it is to understand that a feed-forward ANN is basically just a really fancy transformation matrix.<p>OK, so unless you know linear algebra, you're probably now asking what's a transformation matrix? Without the full explanation, the important understanding is why they are so important in 3D graphics: they can perform essentially arbitrary operations (translation, rotation, scaling) on points/vectors. Once you have set up your matrix, it will dutifully perform the same transformations on every point/vector you give it. In graphics programming, we use 4x4 matrices to perform these transformations on 3D points (vertices) but the same principle works in any number of dimensions - you just need a matrix that is one bigger than the number of dimensions in your data*.<p>Edit: For NNs the matrices don't always have to be square. For instance you might want your output space to have far fewer dimensions that your input. If you want a simple yes/no decision then your output space is one-dimensional. The only reason the matrices are square in 3D graphics is because the vertices are always 3-dimensional.<p>What a neural network does is take a bunch of "points" (the input data) in some arbitrary, high number of dimensions and performs the same transformation on all of them, so as to distort that space. The reason it does this is so that the points go from being some complex intertwining that might appear random or intractable, into something where the points are linearly separable: i.e., we can now draw a series of planes in between the data that segments it into the classifications we care about.<p>The only difference between a transformation matrix and a neural network is that a neural network has at least two layers. In other words, it is two (or more) transformation matrices bolted together. For reasons that are a bit too complex to get into here, allows an NN to perform more complex transformations than a single matrix can. In fact, it turns out that an arbitrarily large NN can perform any polynomial-based transformation on the data.<p>The reason this is often seen as somewhat magic is that although you can tell what transformations a neural network is doing in trivial cases, NNs are generally used where the number of dimensions is so large that reasoning about what it is doing is difficult. Different training methods can give wildly different networks that seemingly give much the same results, or fairly similar networks that give wildly different results. How easy it is to understand the various convolutions that are taking place rather depends on what the input data represents. In the case of computer vision it can be quite easy to visualise the features that each neuron in the hidden layer is looking for. In cases where the data is more arbitrary, it can be much harder to reason about, so if your training algorithm isn't performing as you'd like, it can be difficult to understand why it isn't working, even if you already understand that the basic principle of a feed-forward network is just a bunch of simple algebra.
<a href="http://www.dkriesel.com/en/science/neural_networks" rel="nofollow">http://www.dkriesel.com/en/science/neural_networks</a> about 200 page bilingual ebook about neural networks containing lots of illustrations, in case some guys of you want to read further. There's also a nice java framework to try out for the coders.