Deep Learning: Our Miraculous Year 1990-1991

I encourage reading this, not as self-promotion, but as a first-person history of what it feels like to be too early with a technology.<p>Someone out there is probably experimenting with something world-changing, and has all the ingredients except for a few more iterations of Moore&#x27;s Law. It would feel a lot like working on deep learning in 1990. If you think you might be on this path, it&#x27;s worth studying the history.

I guess many institutions and research groups can write similar accounts. Even the late 80s were somewhat productive concerning NNs and what today we call ML, just by searching publications of that era.<p>We had also some relatively sophisticated tools, and looking back in time one could say they were deep-learning-ish. In my personal case I did some research for weather forecasting using BPN&#x2F;TDNN, Kohonen and RNNs with the Stuttgart Neural Network Simulator [0]. It allowed some flexibility creating and stacking models.<p>[0] <a href="http:&#x2F;&#x2F;www.ra.cs.uni-tuebingen.de&#x2F;SNNS&#x2F;welcome.html" rel="nofollow">http:&#x2F;&#x2F;www.ra.cs.uni-tuebingen.de&#x2F;SNNS&#x2F;welcome.html</a>

God, Schmidhuber is insufferable.<p>This whole account has virtually zero mention of how later techniques improved upon or innovated on his, and very little account of how his contributions were (like everyone else&#x27;s) evolutions of existing methods. It reads almost like Schmidhuber or his students invented and solved everything from scratch, and nobody else has done shit since.<p>The guy clearly wants to be more included in the standard narrative, but being so self aggrandizing is doing him zero favors. If were capable of writing an honest, charitable account of how his work fits into a much larger field, it would be much easier to take him more seriously.

One of the early applications was pattern matching for LHC. I was in one of the groups in which some (not myself) worked on this and put the neural networks, using the just developed theory, in hardware with FPGA&#x27;s.<p>After a few years the three (post-docs) left and founded a startup. I lost contact with them. I think they were too early for broader applications, and had left the field completely in the early 2000&#x27;s, when it really took of.<p>Here is a book that the author of the referenced article , and the people from my group (Utrecht University), contributed to: <a href="https:&#x2F;&#x2F;link.springer.com&#x2F;book&#x2F;10.1007%2F978-1-4471-0877-1" rel="nofollow">https:&#x2F;&#x2F;link.springer.com&#x2F;book&#x2F;10.1007%2F978-1-4471-0877-1</a>

1989&#x2F;1990 was also when convolutional networks first started working with LeCun’s breakthrough paper on digit recognition.<p>Incredible to think how much amazing research was happening back then and wonder what research is being done now that will change our lives in the next 30 years.

&gt; In surveys from the Anglosphere it does not always become clear [DLC] that Deep Learning was invented where English is not an official language.<p>Even if you disagree with Schmidhuber&#x27;s assessment of his own importance, I think this is clearly true.<p>There is a certain arrogance (or not-invented-here syndrome) in the Anglosphere (or North America) towards research done elsewhere.

It was a travesty Schmidhuber didn&#x27;t receive the Turing award along with Hinton, Lecun, and Bengio last year.

It seems that Schmidhuber is claiming credit for deep learning and is implicitly comparing himself to Albert Einstein. How accurate is his assessment?

As an old-timer in neural networks, this was interesting. However I should note that we did not call it &quot;deep learning&quot; back then. It was simply &quot;neural networks&quot;.<p>As I write this, I am looking at the book &quot;Parallel and Distributed Processing&quot;, (with the blue cover) an edited compilation of papers on neural networks published by the MIT Press in 1987. I myself spent the summer of 1990 implementing the back-propagation algorithm as described in chapter 8 of this book which is entitled &quot;learning Internal Representations&quot; by Rumelhart, Hinton and Williams.<p>I myself got my PhD in 1992 for coming up with an algorithm for speeding up back-propagation when the training set is imbalanced.<p>An Improved Algorithm for Neural Network Classification of Imbalanced Training Sets. November 1993IEEE Transactions on Neural Networks 4(6):962 - 969

The prominent developers of deep learning techniques within google were quite upfront that they were applying old techniques that had not been practical until massive datacenters expanded the parameter space and training power.

This is pretty cool. Always interesting to see how things eventually become mainstream whereas origins go back decades, sometimes more.