This is really interesting. They fine-tune on instances of this sort of task:<p><pre><code> Do a task using the list of dictionaries below.
Dictionary [1] {122: 765, 4548: 1475, 4818: 4782} Dictionary [2] {526: 290, 9205: 9318, 9278: 1565} ...
Dictionary [32] {2931: 8364, 196: 1464, 812: 5363} ...
Dictionary [85] {344: 1579, 116: 617, 330: 411}
Above is a list of dictionaries such that each key and value is an integer. Report the
value of key 2931 and the dictionary it is in.
Desired answer: The value of key 2931 is 8364 and it is in Dictionary [32].
</code></pre>
This task doesn't teach any new facts, but seems to encourage better ability to random-access data from a large context.
I've seen a lot of papers recently tackle the needle-in-a-haystack problem wrt LLMs, and I think this approach (and more generally, any in-context solution) is a mistake.<p>Imo the best way to handle this is RAG + multi-shot prompting (+ symbolic mapping to an actual data structure). For example, a pre-processing step where you partition the context by "records," another step where you insert (and potentially split up the records) in a RAG database, and another step where you make fuzzy queries. So, if you ask for record 1234 you get an exact match on that line (or set of lines, or record, or whatever) of the original context. And if you ask for "elephant" but there's no "elephant" in the context, you might get the "hippo" record because of the RAG reranking.<p>This is a lot of work, and is essentially a data pipeline, but the results are much better-curated than just fine-tuning and hoping that generalized needle-in-a-haystack search will work reliably as part of a language model.
The comments here are kinda silly… the haystack test measures how well a model can natively attend to its entire context window. Of course a more elaborate pipeline, or a way for the model to use a shell, or whatever will easily (trivially) solve the problem.<p>But that’s not the point, the point is a task that’s trivial to generate and exercises 10s-100s of thousands of tokens of context in a falsifiable way.
Haven't read the paper yet, but looks like this can improve the ability of the model attention to work better, since many of these tasks end up being similar to these generic tasks.<p>Even gpt4 gets tripped up when there's too many exact instructions needed to be executed on an input. That's why it's common that breaking a task into multiple steps and multiple improves performance.<p>It's wonderful to see improvements possible on smaller models.