Transformers from Scratch

This is a _great_ article. One of the things I enjoy most is finding new ways to understand or think about things I already feel like I know. This article helped me do both with transformer networks. I especially liked how explicitly and simply things were explained like queries, keys, and values; permutation equivariance; and even the distinction between learned model parameters and parameters derived from the data (like the attention weights).<p>The author quotes Feynman, and I think this is a great example of his concept of explaining complex subjects in simple terms.

And here I was, excited to learn something about actual transformers, something involving wire and metal..

This is the best article I have read so far explaining the transformer architecture. The clear and intuitive explanation can’t be praised enough.<p>Note that the teacher has a Machine Learning course with video lectures on youtube that he references throughout the article : <a href="http:&#x2F;&#x2F;www.peterbloem.nl&#x2F;teaching&#x2F;machine-learning" rel="nofollow">http:&#x2F;&#x2F;www.peterbloem.nl&#x2F;teaching&#x2F;machine-learning</a>

This man was my professor at the VU.<p>Honestly his lectures were fun and easy to look forward too, I&#x27;m really glad his post is getting traction.<p>If you find his video lectures they are a really graceful introduction to most ML concepts.

Stellar article, I never understood self attention; this makes it so very clear in a few concise lines, with little fluff.<p>The author has a gift for explaining these concepts.

This is sweet. I&#x27;ve written conv, dense, and recurrent networks from scratch. Transformers next!<p>Plug: I just published this demo using GD to find control points for Bezier Curves: <a href="http:&#x2F;&#x2F;nhq.github.io&#x2F;beezy&#x2F;public&#x2F;" rel="nofollow">http:&#x2F;&#x2F;nhq.github.io&#x2F;beezy&#x2F;public&#x2F;</a>

Ah yes, machine learning architecture transformers, I knew that.

Wow. I have been looking for a good resource on implementing self-attention&#x2F;transformers on my own for the last week - can&#x27;t wait to read this through.

Noob question: I have some 1D conv net for financial time series prediction. Could a transformer architecture be better for this task, is it worth a try?

The title is deceiving. I thought this was an article about building your own electrical transformer, or building your own version of the 1980s toy.