Single Headed Attention RNN

The writing style is amusing. :)<p>Some notes from a first glance:<p>* In the experiments, I see that he uses the Single Headed Attention model actually also with 4 heads, which is kind of a contradiction to the name, isn&#x27;t it?<p>* The main motivation is performance (training speed mostly). So some absolute number of e.g. training time would be nice to have in the comparisons. He e.g. mentions that the Adaptive Transformer can also be trained on a single GPU within hours, and in the comparison, the Adaptive Transformer gets much better BPC (enwik8), and uses even slightly less parameters. So, isn&#x27;t the Adaptive Transformer thus better in every aspect (speed and BPC)? Or how does it compare in speed? As far as I remember, also the Sparse Transformer is more efficient (as it has sparsity), so again the speed comparison would be interesting here. Or is the argumentation for inference speed? But then the inference speed should be compared, or not?

Honestly, I wish all research papers were written this way. Easy to understand, kept me entertained, and presented meaningful results with a way to reproduce (on a single GPU).<p>I grant all research papers on deep learning can&#x27;t be reproducible with a single GPU in a reasonable time, but it should happen more often IMO. It seems lazy to just toss out a paper saying &quot;we hit new benchmarks, by increasing the parameters and throwing more compute&quot;. I&#x27;d like to see &quot;we hit new benchmarks with a new design, the old ones had this issue, etc.<p>Anyway, great read, recommend. Also, happy for the author haha<p>&quot;The author has also moved to a one bedroom apartment in San Francisco, removing themselves from proximity to the alley of questionable odors and unsavory noises.&quot;

for those who aren&#x27;t familiar with the author, he previously worked at metamind &#x2F; salesforce research doing nlp and has published many successful nlp papers [0]. he opted to write an informal paper for this project (similar to yolov3 [1]), but the work itself should still be taken seriously.<p>[0] <a href="https:&#x2F;&#x2F;scholar.google.com&#x2F;citations?user=AolIi4QAAAAJ" rel="nofollow">https:&#x2F;&#x2F;scholar.google.com&#x2F;citations?user=AolIi4QAAAAJ</a><p>[1] <a href="https:&#x2F;&#x2F;pjreddie.com&#x2F;media&#x2F;files&#x2F;papers&#x2F;YOLOv3.pdf" rel="nofollow">https:&#x2F;&#x2F;pjreddie.com&#x2F;media&#x2F;files&#x2F;papers&#x2F;YOLOv3.pdf</a>

You just have to love Stephen Merity.<p>His work on QRNN&#x27;s saved me quite a bit of time and money when I was doing my undergrad dissertation on language models.<p>This SHA-RNN seems to have surfaced from a similar line of thinking that spawned the QRNN.

The paper rises a great point on tokenization affecting perplexity, that we can&#x27;t compare perplexities of different tokenizers even re-normalizing taking token counts into account, say BPE vs word tokenization. This example nails it: <a href="https:&#x2F;&#x2F;twitter.com&#x2F;Smerity&#x2F;status&#x2F;1192252147598909441" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;Smerity&#x2F;status&#x2F;1192252147598909441</a>

Perhaps I am missing the point of this article. The RNN approach seems to get similar performance, but uses more parameters and misses the parallelization benefits that Transformers have and recurrent networks do not.<p>What is the benefit of the RNN here?

Another work in the opposite direction, introducing gating in Transformer-xl <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;1910.06764" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;1910.06764</a>

Hilarious papers, i&#x27;m about to drop a SHA-RNN on my GPu to make it sweat

Did anyone one else read &quot;SHA-RNN&quot; as &quot;SHHHAAAAARRRROOOOONNN&quot; in Ozzy&#x27;s voice?

Now that was some refreshing reading.

ok

Now I really want pop music made by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin

A dissenting voice from the positive reception here on HN, I thought that this paper was a joke. Single author, no affiliation, snarky language. Why not be civil instead?