This was originally a LessWrong post [0], but I linked to the colab for its interactivity, as the authors advise:<p>"Please go to the colab for interactive viewing and playing with the phenomena. For space reasons, not all results included in the colab are included here so please visit the colab for the full story."<p>[0] <a href="https://www.lesswrong.com/posts/mkbGjzxD8d8XqKHzA/the-singular-value-decompositions-of-transformer-weight" rel="nofollow">https://www.lesswrong.com/posts/mkbGjzxD8d8XqKHzA/the-singul...</a>