26× Faster Inference with Layer-Condensed KV Cache for Large Language Models

Upshot of the paper -- right now KV caches are implemented for multiple deep layers in LLMs. Why not just the top layer? It would save memory.<p>Initial result -- those KV caches in lower layers matter, and output suffered.<p>Updated plan -- cull half the KV layers! This works &#x27;nearly&#x27; as well as keeping all of them, with memory and compute savings.<p>Downside - triple the training, worse out of band &#x2F; long context performance.<p>This feels to me like a technique you&#x27;d use on a particular architecture deployed at the edge where compute matters and you have a little extra room on performance. Phi-3 on raspberry pi, basically.<p>Interesting! As always, I wish models showed prompt output in their papers, not just perplexity numbers. But, here we are.

Not sure if @dang is the right way to say the title is incorrect here, but shouldn&#x27;t it match the paper?<p>1. The correct title is Layer-Condensed KV Cache for Efficient Inference of Large Language Models.<p>2. The paper does make a 26x claim later in the introduction, but it’s an outlier.<p>26x is for only one benchmark and that benchmark is CPU based, not GPU based like 99% of transformer loads actually run on.<p>If you look at GPU only workloads, the improvements range from 1.4x to 4.7x.

LLM inference optimization has been key for the OpenAI GPT-4o presentation (2x faster, 50% cheaper) and its driving lots of industry research because it’s direct cost savings, but it’s refreshing to see so many techniques published as papers (i.e from Stanford, Berkeley…)

&gt; please use the original title, unless it is misleading or linkbait; don&#x27;t editorialize.<p>&quot;Layer-Condensed KV Cache for Efficient Inference of Large Language Models&quot;

&quot;Our implementation is based on HuggingFace transformers where we register a new model opt-llama that supports the Layer-Condensed KV Cache.&quot;<p>Not sure what this means? Would this work for a Mistral model as well?

This can be combined with Grouped Query Attention or Multi-Query Attention for an even further reduction in the size of the KV Cache!

Is the KV cache something that runs on the GPU or on the CPU? Or traditionally on the CPU &amp; this enables it to run on the GPU?