That is an excellent explanation full of great intuition building!<p>If anyone is interested in a kind of tensor network-y diagrammatic notation for array programs (of which transformers and other deep neural nets are examples), I wrote a post recently that introduces a kind of "colorful" tensor network notation (where the colors correspond to axis names) and then uses it to describe self-attention and transformers. The actual circuitry to compute one round of self-attention is remarkably compact in this notation:<p><a href="https://math.tali.link/raster/052n01bav6yvz_1smxhkus2qrik_0736_0884_02kdqvrzq963t.jpg" rel="nofollow noreferrer">https://math.tali.link/raster/052n01bav6yvz_1smxhkus2qrik_07...</a><p>Here's the full section on transformers: <a href="https://math.tali.link/rainbow-array-algebra/#transformers" rel="nofollow noreferrer">https://math.tali.link/rainbow-array-algebra/#transformers</a> -- for more context on this kind of notation and how it conceptualizes "arrays as functions" and "array programs as higher-order functional programming" you can check out <a href="https://math.tali.link/classical-array-algebra" rel="nofollow noreferrer">https://math.tali.link/classical-array-algebra</a> or skip to the named axis followup at <a href="https://math.tali.link/rainbow-array-algebra" rel="nofollow noreferrer">https://math.tali.link/rainbow-array-algebra</a>