Copy is all you need

I think the &quot;... is all you need&quot; title here is particularly misleading as the paper does in fact use a BERT model for generating the vectors.<p>So if the implication was that no language model was needed at all and you can just do nearest neighbour on string similarity and patch results together, that implication was clearly wrong.<p>I think what the paper does show though is that there are methods that can make language models topic-specific without fine-tuning and that yield competitive results even with older models.

This resonates with the current AI skeptic view that language models are a supercharged search engine on the pile of text they&#x27;re trained on.<p>Also the fact that evaluating language models is difficult, and we tend to end up with models that game the evaluation benchmarks.

What about LLM reasoning ability?<p>Faith and Fate: Limits of Transformers on Compositionality <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2305.18654" rel="nofollow noreferrer">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2305.18654</a><p>Transformers solve compositional reasoning tasks by reducing multi-step compositional reasoning into linearized subgraph matching without problem-solving skills. They can solve problems when they have reasoning graphs in the memory.

Seems like a common pattern. State of the art models being well replaced by a information retrieval layer (top 10 results) fed into a much lighter model that does something with that plus the original input. Cool result!

This approach can probably handle most of the queries search engines and Siri-type chatbots handle. The big GPT-type engines can be reserved for the hard problems. Something along those lines is needed to keep the cost of search down. There&#x27;s an estimate that using a large language model for search is 10x more expensive than existing search engines. Yet few queries really need that much heavy machinery.

The big advantage here would be the ability to attribute entire blocks of text back to a specific source and cross domains just by building a database of embeddings. The downside is that these networks are probably not as creative as they&#x27;re limited to only data that&#x27;s available. It might work best to use something like this as an expert system for a GPT like agent to refer to when needed.

Obvious immediate question is, is it as creative? There are a lot creativity left behind when you increase the token size (let&#x27;s be real, it&#x27;s just that). As an example creating a new word like &quot;dickstracted&quot;[1] would not ever happen in this model<p>[1] <a href="https:&#x2F;&#x2F;www.urbandictionary.com&#x2F;define.php?term=Dickstracted" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.urbandictionary.com&#x2F;define.php?term=Dickstracted</a>

Slight tangent:<p>I once worked with a programmer who, the vast majority of time, would only input text into a text editor via copy and paste.<p>Think anti-vim. His fingers were locked on mouse and crtl+c&#x2F;v. It was incredible to watch and his programming speed was very impressive.

This has deep connections with my attempt to implement an effective queryable word-level grammatically correct extractive text summarizer (AKA: The way most people actually summarize documents) - <a href="https:&#x2F;&#x2F;github.com&#x2F;Hellisotherpeople&#x2F;CX_DB8">https:&#x2F;&#x2F;github.com&#x2F;Hellisotherpeople&#x2F;CX_DB8</a><p>I will try to implement this with the necessary changes to actually make this work properly, where instead of generating a new answer, it simply highlights the most likely text spans.

Surprised no one has mentioned the obvious issue: plagiarism<p>(Not sure if the authors have indicated any method for attribution of the original data)

this made me think of a fun activity. ask chatgpt to come up with a new word and then google that word. sometimes the word exists in the context of a scify show or a plant. sometimes gpt just added a &quot;se&quot; or &quot;us&quot; to existing words. sometimes it changes a Z to a C but it never actually came up with a new word

Behold, the <i>true</i> stochastic parrot.

This is interesting coming on the heels of the gzip-based inference paper. gzip is based on LZ77, and the LZ family of compressors generate and store (and cleverly encode) instructions to copy blocks of text they have seen before to their output.

I remember that around 2004, before convnets became popular, there was a paper on image texture style transfer using approximate nearest neighbors based on some neighborhood of each point. This technique seems similar but for text.

&gt; COG<p><a href="https:&#x2F;&#x2F;wiki.opencog.org&#x2F;w&#x2F;The_Open_Cognition_Project" rel="nofollow noreferrer">https:&#x2F;&#x2F;wiki.opencog.org&#x2F;w&#x2F;The_Open_Cognition_Project</a>

Thought this was about how you just need good copywriting skills

I wish we could stop with the “X is all you need” papers! The first one was unintuitive and so are the rest.

First, hate the title<p>Second, this approach seems equivalent to using larger tokens, which means the problems with using tokens instead of letters are just exacerbated