Hopfield Networks Is All You Need

Brief abstract for the lay person (like me):<p>1. Hopfield Networks are also known as &quot;associative memory networks&quot;, a neural network model developed decades ago by a guy named Hopfield.<p>2. It&#x27;s useful to plug these in somehow as layers in Deep Neural Networks today (particularly, in PyTorch).<p>I hate non-informative titles!

Trending as John Hopfield scheduled to present his &quot;biologically plausible&quot; response to the Modern Hopfield Network at ICLR next week:<p>Large Associative Memory Problem in Neurobiology and Machine Learning<p><a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2008.06996" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2008.06996</a><p>MHN seem ideal for prediction problems based purely on data, such as chemical reactions and drug discovery:<p>Modern Hopfield Networks for Few- and Zero-Shot Reaction Prediction<p><a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2104.03279" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2104.03279</a>

“Sooner or later, everything old is new again.” -Steven King

Quoting: &quot;We introduce a new energy function and a corresponding new update rule which is guaranteed to converge to a local minimum of the energy function.&quot;<p>Is this a minimum in a local area or local in the range of some function? I could see perhaps that&#x27;d being an advantage if you happen to know that local part of the range<p>In contrast we&#x27;re usually looking for global min&#x2F;max say with annealing algorithms. How is local is better in the context of this paper than global?

I’ve seen a lot of efforts to add a notion of associative memory into neural networks. Have any exciting applications of such architectures been publicised?

I looked at the paper but it was way over my head.<p>Can anyone explain it in simpler terms to a person who barely understands attention models and has no idea what associative memory means here?

Nice paper! I used Hopfield networks in the 1980s. I hope that I can clear a few hours of time this week to work through this. I admit that for machine learning, that I have fell into the “deep learning for everything pit” in the last six or seven years. Probably because DL is what I usually get paid for.

Off-topic, but does anyone know what Jekyll theme this is? Absolutely beautiful formatting and color scheme.

This reminded me of a very old fun side project of mine [1] that had made me look at neural networks from a different perspective.<p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;milosgajdos&#x2F;gopfield" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;milosgajdos&#x2F;gopfield</a>

If I understood them correctly, they store all the training samples and then select one most similar to a given input.

Are*

Amazing