Real-Time Introspective Compression for Transformers

Cool! On a quick glance, it doesn't seem like the group/layer index is provided to the compression model. That might help a bit with fidelity at pretty low additional cost.

Strategy distillation seems like gradient update in a way? Or would that be at a higher abstract level.

Gemini Pro thinks this is a April 1 prank and says the computational cost makes it infeasible.

Isn't that basically a recurrent neural network?

I recently started thinking about what a shame it is that LLMs have no way of directly accessing their own internal states, and how potentially useful that would be if they could. One thing led to the next, and I ended up developing those ideas a lot further.<p>Transformers today discard internal states after each token, losing valuable information. There&#x27;s no rollback, introspection, or replaying of their reasoning. Saving every activation isn&#x27;t practical; it would require way too much space (hundreds of megabytes at least).<p>The insight here is that transformer activations aren&#x27;t randomly scattered in high-dimensional space. Instead, they form structured, lower-dimensional manifolds shaped by architecture, language structure, and learned tasks. It&#x27;s all sitting on a paper-thin membrane in N-space!<p>This suggested a neat analogy: just like video games save compact states (player location, inventory, progress flags) instead of full frames, transformers could efficiently save &quot;thought states,&quot; reconstructable at any time. Reload your saved game, for LLMs!<p>Here&#x27;s the approach: attach a small sidecar model alongside a transformer to compress its internal states into compact latent codes. These codes can later be decoded to reconstruct the hidden states and attention caches. The trick is to compress stuff a LOT, but not be TOO lossy.<p>What new capabilities would this enable? Transformers could rewind their thoughts, debug errors at the latent level, or explore alternative decision paths. RL agents could optimize entire thought trajectories instead of just outputs. A joystick for the brain if you will.<p>This leads naturally to the concept of a rewindable reasoning graph, where each compressed state is a node. Models could precisely backtrack, branch into alternate reasoning paths, and debug the causes of errors internally. Like a thoughtful person can (hopefully!).<p>Longer-term, it suggests something bigger: a metacognitive operating system for transformers, enabling AI to practice difficult reasoning tasks repeatedly, refine cognitive strategies, and transfer learned skills across domains. Learning from learning, if you will.<p>Ultimately, the core shift is moving transformers from stateless text generators into cognitive systems capable of reflective self-improvement. It&#x27;s a fundamentally new way for AI to become better at thinking.<p>For fun, I wrote it up and formatted it as a fancy academic-looking paper, which you can read here:<p><a href="https:&#x2F;&#x2F;raw.githubusercontent.com&#x2F;Dicklesworthstone&#x2F;llm_introspective_compression_and_metacognition&#x2F;main&#x2F;introspective_compression_for_llms.pdf" rel="nofollow">https:&#x2F;&#x2F;raw.githubusercontent.com&#x2F;Dicklesworthstone&#x2F;llm_intr...</a>