Overpromises and under delivers.<p>> Hebbian learning, can implement a transformer-like attention computation if the synaptic weight changes are large and rapidly induced<p>Well this has no analog in the brain either.<p>In any case. How is this transformer like attention? Not all attention mechanisms lead to a transformer. Certainly the two are not synonymous.<p>> While it is tempting to assume that cortical rhythms might play a role, we have found that match windows around 0.5—2 s are necessary for reducing noise in the spike train comparisons, a timescale much longer than the cycles of theta, beta or gamma rhythms found in the cortex.<p>Well that's totally the wrong timescale.<p>There's a glut of these "abuse some random feature of the brain that already serves a different purpose to implement something that clearly doesn't work but is vaguely reminiscent of something that happens in machine learning so we'll call the two the same". They contribute nothing.<p>The few somewhat worthwhile actually show a working network for a real task. The real breakthrough will be a paper that actually works for real, on real data, and can be implemented in the brain; we've got nothing like that. This isn't one of the good ones.
This is not surprising considering the paper "Hopfield Networks is All You Need" <a href="https://arxiv.org/abs/2008.02217" rel="nofollow">https://arxiv.org/abs/2008.02217</a>. Hopfield networks learn via Hebbian learning.