Some interesting ideas sadly blocked behind a pay-wall journal, all for the purpose of boosting a researcher's prestige because they now hold a "Nature" publication. Thankfully, this article is easily accessible via Sci-Hub. <a href="http://www.nature.com.sci-hub.cc/nature/journal/vaop/ncurrent/full/nature20101.html" rel="nofollow">http://www.nature.com.sci-hub.cc/nature/journal/vaop/ncurren...</a>
This is probably the most important research direction in modern neural network research.<p>Neural networks are great at pattern recognition. Things like LSTMs allow pattern recognition through time, so they can develop "memories". This is useful in things like understanding text (the meaning of one word often depends on the previous few words).<p>But how can a neural network know "facts"?<p>Humans have things like books, or the ability to ask others for things they don't know. How would we build something analogous to that for neural network-powered "AIs"?<p>There's been a strand of research mostly coming out of Jason Weston's Memory Networks research[1]. This extends on that by using a new form of memory, and shows how it can perform at some pretty difficult tasks. These included graph tasks like London underground traversal.<p>One good quote showing how well it works:<p><i>In this case, the best LSTM network we found in an extensive hyper-parameter search failed to complete the first level of its training curriculum of even the easiest task (traversal), reaching an average of only 37% accuracy after almost two million training examples; DNCs reached an average of 98.8% accuracy on the final lesson of the same curriculum after around one million training examples.</i><p>[1] <a href="https://arxiv.org/pdf/1410.3916v11.pdf" rel="nofollow">https://arxiv.org/pdf/1410.3916v11.pdf</a>
Blog post for the paper: <a href="https://deepmind.com/blog/differentiable-neural-computers/" rel="nofollow">https://deepmind.com/blog/differentiable-neural-computers/</a>
Very exciting extension of Neural Turing Machines. As a side note: Gated Graph Sequence Neural Networks (<a href="https://arxiv.org/abs/1511.05493" rel="nofollow">https://arxiv.org/abs/1511.05493</a>) perform similarly or better on the bAbI tasks mentioned in the paper. The comparison to existing graph neural network models apparently didn't make it into the paper (sadly).
Can someone explain what the full implications of this are? This seems really cool, but I can't really wrap my head around it.<p>From what I can tell you can give the DNC simple inputs and it can derive complex answers.
if you're interested in this check out "Reasoning, Attention, Memory (RAM)" NIPS Workshop 2015 organized by Jason Weston (Facebook Research): <a href="http://www.thespermwhale.com/jaseweston/ram/" rel="nofollow">http://www.thespermwhale.com/jaseweston/ram/</a>
I have a couple questions that I'm not getting from this, does this memory persist between each "instance" of a task? Or does it get wiped out after each one? Is this something where you might say present the model with some data that is the input (which it might learn to then store in memory) and then ask a question of it?<p>i.e, in the blog post it discusses using the network to find the shortest path between two stations, would the steps to do that look like this?<p>1. Train the NN how to navigate any network, presenting the graph data each time you ask the NN a problem
2. take the trained NN and feed it the London Underground, then ask it to tell you how to get there?
Instead of saving the data, you could think of using a memory address as applying the identity function and saving the data.<p>Could it learn to use addresses that perform more interesting functions than f(x)=x?
I'm probably totally off base here (neural networks/AI is not my wheelhouse), but is having "memory" in neural networks a new thing? Isn't this just a different application of a more typical 'feedback loop' in the network?
> a DNC can complete a moving blocks puzzle in which changing goals are specified by sequences of symbols<p>A neural network without memory can't do that or can't do it as well perhaps?