Vision Transformers Need Registers

[2023]<p>Previously:<p>2 years ago: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=37794996">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=37794996</a>.<p>1 year ago: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=40329675">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=40329675</a>

Extremely cool!<p>It&#x27;s interesting and honestly encouraging that this kind of thing can be discovered and understood using just &quot;simple linear methods&quot; and high-level analysis of patterns in layer activations.

So basically multiple CLS tokens.<p>Fwiw, I tried multiple global tokens in my chess neural net and didn&#x27;t see any uplift compared to my baseline of just having one.

Has this been used widely since?