Discussion & links to various implementations: <a href="https://www.reddit.com/r/MachineLearning/comments/eg1wr3/reformer_the_efficient_transformer_anonymous_et/" rel="nofollow">https://www.reddit.com/r/MachineLearning/comments/eg1wr3/ref...</a>
There is no argument for why the LSH would work well, especially at the beginning of training. As the weights are initially random, bucket assignment would be random as well. If predicting at position A requires info from position B, but they are not in the same bucket, there will be no gradient to get the query embedding of A closer to the key embedding of B. The reversible layer trick is neat though.
One neat trick is that you can extend GPT-2 117M's context window from 1024 up to 30k on a TPU, since TPUs can allocate up to 300GB of memory for backprop. <a href="https://twitter.com/gwern/status/1218001309435072513" rel="nofollow">https://twitter.com/gwern/status/1218001309435072513</a><p>It's not quite 1M words, but a 30k context window is big enough for e.g. most midi songs.
This seems like a big deal. An asymptotic reduction in the resource explosion created by larger attention windows should allow the development of substantially more complex models here.
Vowpal Wabbit has been doing this 'hashing trick' since the 200s.<p>It also the feature interaction, which are the same thing as a layer in transformers (all against all matrix).<p>So it seems like they are still catching up to where John Langford and crew were over a decade ago.<p>And, the vowpal wabbit approach is extremely fast to train because it's only doing stochastic gradient descent on a linear function - linear regression. Transformers are much slower to train.<p>EDIT: Downvoters, please see my last leaf to see why they're effectively the same. The guy responding here seems unfamiliar with all the functionality of vowpal wabbit.