I think people are misreading this work, and assuming this is equivalent to full dense-attention. This is just saying its an efficiency gain over sliding window re-computation, where instead of computing the L^2 cost over and over (T times), you can re-use a cache and maintain perplexity. I don't think they are claiming that this allows for attending to content that was far away.<p>They tested by running concatenating and measuring -> `Q A Q A Q A Q A...` not by doing `Q Q Q Q A A A A...`<p>They also measure perplexity, showing that it produces "readable text" (coherent, locally viable); not that it is "extracting anything" from the big-triangle-gap of no-attention.<p>I think this would fail to be given a book, then write the first word of every paragraph. Or, given a book, write a 1 sentence summary of each chapter. I might be wrong, because they didn't test tasks like this, but I'd be very very surprised.
On a first quick pass, this looks so good that I'm wondering if it's <i>too good to be true</i>!<p>But the work looks to be of decent quality and the technique is remarkably straightforward:<p>The idea is to apply attention over the first token and a sliding context window, ignoring everything in-between, in each layer.<p>By implication, each layer must be gradually shifting relevant information forward in the sequence, enabling the top layer's ending sliding attention window to see it.<p>The only caveat I can think of is that the sliding windows won't be able to shift all important information forward when the span of all sliding windows isn't sufficient to span the entire sequence -- for example, when model depth × window length < sequence length, if all windows have the same length.
This seems to be largely enabled by the observation that Softmax has to add up to one. From quick a glance [1], the model tends to use the first token as a placeholder for cases when you don't need to attend any of the prior tokens.<p>The first time I read about this issue, that Softmax is somewhat flawed, was in a HN post by Evan Miller [2] where he observes that forcing attention heads to allocate all attention to prior tokens is wrong, and we should allow them to "not attend" by adding one to the softmax denominator.<p>I love that they found a way to capitalize on this observation without having to retrain models. However, I wonder how the models would look like if they followed Evan's suggestion!<p>[1] Their description of attention sinks:<p>```<p>To understand the failure of window attention, we find an interesting phenomenon of autoregressive
LLMs: a surprisingly large amount of attention score is allocated to the initial tokens, irrespective
of their relevance to the language modeling task, as visualized in Figure 2. We term these tokens
“attention sinks". Despite their lack of semantic significance, they collect significant attention scores.
We attribute the reason to the Softmax operation, which requires attention scores to sum up to one
for all contextual tokens. Thus, even when the current query does not have a strong match in many
previous tokens, the model still needs to allocate these unneeded attention values somewhere so it
sums up to one. The reason behind initial tokens as sink tokens is intuitive: initial tokens are visible
to almost all subsequent tokens because of the autoregressive language modeling nature, making
them more readily trained to serve as attention sinks.<p>```<p>[2] <a href="https://news.ycombinator.com/item?id=36851494">https://news.ycombinator.com/item?id=36851494</a>
Adding attention cache memory is an extremely interesting solution to this problem.<p>If anyone is curious, there was another paper [0] that came out a few days ago that made a related observation in Vision Transformers. Transformer models appear to pick tokens to store global information in - they need tokens to "think". You can eek some performance improvements (and cool explanation images) by providing the model with specific tokens for this purpose.<p>[0] <a href="https://arxiv.org/pdf/2309.16588.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/2309.16588.pdf</a>
The authors just uploaded a FAQ section, which may clarify some of the confusions: <a href="https://github.com/mit-han-lab/streaming-llm/blob/main/README.md#faq">https://github.com/mit-han-lab/streaming-llm/blob/main/READM...</a>
I could be wrong, but I'm not sure this is about what people seem to think it is, e.g., letting LLMs reference content past the trained length<p>I think it may just be about the performance of the model with longer texts (on the things still within the context window?). It sounds like they're arguing that the model is essentially learning to stick some baggage in the attention to the initial tokens of the text, and break when that isn't within the window anymore for reasons I'm not sure I understand (after all, isn't text in the middle just as good as text at the start for non instruction inputs?)
How do any of these sliding window techniques handle instructions that are non expected and only show up at the end? For example imagine feeding a book to the model and the last sentence being the instruction “return the count of the letter m in the previous input”. A human would handle this by first letting out an exasperated sigh but then restarting the reading while counting. An LLM has no ability to loop back and re-read the input. (Ignore LLM issues with character counting for this example). It seems like to solve this problem for real the LLM needs to be able to loop and jump arbitrarily, but I’m sure that would introduce a whole new host of issues and possibly require a new architecture all together.
Relevant: the eponymous Professor Han at MIT is teaching a TinyML course that is open to the public. See:<p>- <a href="https://news.ycombinator.com/item?id=37620507">https://news.ycombinator.com/item?id=37620507</a><p>- <a href="https://efficientml.ai" rel="nofollow noreferrer">https://efficientml.ai</a>
This looks fantastic. Also answers the relevancy of the "off-by-one" softmax*<p>My naive question is...does it work? But that sounds dismissive. At length:<p>It shows that the model can't respond after a certain length versus a proposed model that does continue to respond.<p>But can a model that continues to respond retrieve information far "in the past"?<p>The demo video is too low-level, at least to my brain. It shows one model stops responding but the proposed one continues.<p>I spent about 5 minutes going frame by frame to see if the proposed model attempts to have to "recall" information from further back, but it looks like no.<p>Perfection here isn't necessary or even possible AFAIK, i.e. I don't expect it to recall page 1 100% accurately at page 1000. But can it recall _anything_ from it, even if it ignores it?<p>The great thing about this era and work is we can check. But I hope someone has it up in a HuggingFace space before I figure out how to run it myself. :P<p>I'm leaning no, based on the sliding window thing. It sounds like there's 4 fixed tokens, then the last context size - 4 tokens, that's it<p>* at the time, two camps: one, it's some random person saying it and there's prior art on implementations that do the off-by-one. Two, you'd be surprised how much little things go unnoticed by large groups, and do matter.
I feel like information theory prevents full information retention for unlimited context lengths and finite compute, but I don't know if we are at information theory limits to invoke this argument. Or rather, I don't know how to make a good analysis of (bits of context information) per (bits of model parameters).
This is working relatively well, the code is really worth a read. If you run it locally, consider the open PR and install sentencepiece as well. It's been generating text for the past 10 minutes now :D<p>Some of the instructions are ignored though so I'd be careful there, one instruction is to rewrite the previous response by "starting every sentence with the letter A" which is a bit of a hit or miss right now.
Is it just me, or does every approach basically boil down to not wanting to pay the full quadratic cost over the context (usually by selecting which tokens to pay attention to, or using some computationally cheaper substitute for each token).<p>I feel like all these approaches kind of equivalent to a fully dense attention matrix over a smaller context, but carefully curating what goes into the context, also known to us humans as summarizing each bit of text, or (perhaps less efficiently) going through a textbook with a highlighter.<p>My intuition is that the winning approach will be a small (ish), lets say 8k context, with efficient an summarization and dynamic information retrieval scheme.
Did anyone ever attempt a recursive architecture?<p>So you take the first window or logical separatation (chapter, paragraph) and let the model summarize it into one or two sentences. Then you repeat that with the next window (and that derived sentence as context) and create a new logical separatation out out of a fixed number of sentences. Rinse and repeat until the result fits into your window.<p>I have a hunch that this is somewhat how the brain works when reading.
Looks like longformer to me. They just renamed "global attention" into "attention sink" and removed silly parts(distilled attention) and BERT parts([CLS] saw all N tokens, there is no need for BOS to see all tokens)
Anyone got a gut feel as to whether you could use this to transform Whisper into a better streaming model? It's a bit of a hack using it that way at the moment.
We introduce StreamingLLM, an efficient framework that enables LLMs trained with a finite length attention window to generalize to infinite sequence length without any fine-tuning. We show that StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more.
What if my "Favorite LLM" is GPT4? I don't want to use Llama or anything like that. Does this GitHub code let me use the OpenAI API and run the new memory technique on top of that?