The Curse of Recursion: Training on Generated Data Makes Models Forget

Ted Chiang predicted this in The New Yorker [1] in February in an article that shaped my thinking about what LLMs are capable of achieving in the near future. Chiang compared the summaries LLMs synthesize to a lossy compression algorithm for the internet.<p>&quot;There is very little information available about OpenAI’s forthcoming successor to ChatGPT, GPT-4. But I’m going to make a prediction: when assembling the vast amount of text used to train GPT-4, the people at OpenAI will have made every effort to exclude material generated by ChatGPT or any other large language model. If this turns out to be the case, it will serve as unintentional confirmation that the analogy between large language models and lossy compression is useful. Repeatedly resaving a jpeg creates more compression artifacts, because more information is lost every time. It’s the digital equivalent of repeatedly making photocopies of photocopies in the old days. The image quality only gets worse.<p>Indeed, a useful criterion for gauging a large language model’s quality might be the willingness of a company to use the text that it generates as training material for a new model. If the output of ChatGPT isn’t good enough for GPT-4, we might take that as an indicator that it’s not good enough for us, either. Conversely, if a model starts generating text so good that it can be used to train new models, then that should give us confidence in the quality of that text. (I suspect that such an outcome would require a major breakthrough in the techniques used to build these models.) If and when we start seeing models producing output that’s as good as their input, then the analogy of lossy compression will no longer be applicable.&quot;<p>[1] <a href="https:&#x2F;&#x2F;www.newyorker.com&#x2F;tech&#x2F;annals-of-technology&#x2F;chatgpt-is-a-blurry-jpeg-of-the-web" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.newyorker.com&#x2F;tech&#x2F;annals-of-technology&#x2F;chatgpt-...</a>

Wouldn’t it be funny to find that the capabilities of LLM models have already peaked because we are unable to restrain ourselves from polluting the internet and other training corpus sources with their output?

Hah, I brought this up here a few months ago and was quickly dismissed.<p>I wonder if opening GPT and DALLE to the public was partly intended to pollute subsequent data for anyone that gets into AI down the road. Suddenly a lot of publicly accessible data is worth less, leaving only players who&#x27;ve got a hoard of time-stamped data to compete with (like Google, Facebook). OpenAI almost certainly has the hashes of what it spits out too, so they&#x27;ll be able to sort the wheat from the chaff for a while yet.<p>The market for data may be getting interesting.

I don&#x27;t believe in the &quot;dead internet theory&quot; as a description of the current situation (mostly), but as a prediction? Maybe.<p><a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Dead_Internet_theory" rel="nofollow noreferrer">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Dead_Internet_theory</a>

This is not a model problem or synthetic data problem. This is common data science and the article says that: &quot;We find that use of model-generated content in training causes irreversible defects in the resulting models, where tails of the original content distribution disappear.&quot; Data quality is more important than data volume and if you forget about that... garbage in, garbage out.<p>Make sure you have a representative training dataset, real or synthetic, it doesn&#x27;t matter.

There is a massive flaw in this argument. In real life, whether a given generated work ends up in a future dataset depends on how good it is according to humans. For example, in order for an article to end up in the reddit set it needs at least three upvotes.<p>They could have replicated it here by having GPT-4 score the samples and throwing out most (but not all) of the bad ones. I have no idea what would happen if you e.g. throw out a majority of the bottom 70% and keep the top 30%. It&#x27;s conceivable to me that it would end up improving or at least not getting much worse with each generation.

Generated data tends to be selected and edited by humans, so it is already a bit better than raw. In general a model that takes external feedback into account will be able to self improve, for example a code model would run tests and a chat model could interpret user responses as reward signals.<p>You gotta add something new to the mix. That&#x27;s possible when the AI is part of a larger system. AlphaZero demonstrated that even self play could be a source of signal, as long as it gets the feedback of the game, the model can learn.

I think one of the things overlooked in the discussions here is that the research is solely around the reinforcement against edge cases, but does not qualitatively assess these edge cases.<p>To me, this research supports a hypothesis I&#x27;ve had for a while that we&#x27;re going to get to truly excellent AI by using synthetic data to bias it towards excellence and away from mediocrity.<p>$20 says the next round of major model training is using synthetic data for training that was filtered through a discriminator trained entirely on human data.<p>The human data as a reference is certainly important to avoid polluting (and to its point there&#x27;s an advantage for those already having it), but moving away from edge cases isn&#x27;t necessarily a bad thing practically given edge cases can result in negative practical performance (as opposed to academic next token performance).

Same idea here? Larger models do a better job forgetting their training data and dropping their semantic priors. Perhaps another way of thinking through this is that larger models learn new information and drop old information faster. <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2303.03846" rel="nofollow noreferrer">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2303.03846</a><p>Isn&#x27;t that interesting? The idea of &quot;mental liquidity&quot;, or &quot;strong opinions weakly held&quot;? <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=36280772">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=36280772</a>

&gt; For the private sector, many homeowners and corporations have longer-term fixed debt, and only some portion of it matures each quarter and gets refinanced at higher rates. As more private debt matures and gets refinanced at higher rates, this will continue to serve as a disinflationary and recessionary force on the economy, especially for sectors that are more sensitive to interest rates.<p>The one thing I don&#x27;t get and could have been missing in the past... a lot of the corporations and private things, like farms operate on debt. Now maybe it&#x27;s a bit reductionist, but if you&#x27;re a farmer operating on debt, if interest rates go up you need to increase prices to cover operating expenses. And this get compounded all the way up to the end consumer as every step in the supply chain marks up by a fixed percent, and because everything is getting more expensive decided lets mark up by a larger percent. So higher interest rates really could be contributing to inflation. And it&#x27;s just creating a cycle. And with the current levels of debt never seen before in history, it&#x27;s unlike other periods.

Side effect: Search engine content if not detected for generated content would be the first to suffer.

Recursive training of generative models degenerates into reinforcement learning with random feedback. You need strong feedback signal in the loop to survive recursive training. The feedback does not have to come from humans though. You can use anything that grounds the model in reality.

Self play in RL is signal enough that machines can learn on their own. How we train models and what class of models is important. No doubt the paper makes good points but I don&#x27;t think the reality is so black-and-white.

I am in way over my head here, so I wasn&#x27;t able to tell if the authors addressed this, but my intuition is that this should be somewhat mitigated so long as people are providing the filter between which results are discarded and which might end up back in the training pool.<p>I would think that the human selection process would help to head off this conversion, both by selecting against incorrect results, and also by introducing variance outside of the model. On the other hand since a person can only act as a filter, I can also see how that would be of limited value long term.

As long as the synthetic data is good, how can you tell the difference between it and human generated data?<p>This paper has one huge hole in it: it assumes that content on the internet is not moderated and that the training dataset will never evolve to take rating into consideration. On social media, the form of moderation is # of likes. Once detected, bots that output bad data will be banned and content deleted.<p>The key issue I have with the paper is for good synthetic data it is impossible to tell it apart from human generated data.

Not surprising. It always seemed likely to me that there is model bias if you train your models on model generated data, like a feedback loop (second order effects?). Similar to how applying a linear system over and over stretches the inputs in the direction of its largest eigenvector.<p>Now wait till the generated content is indistinguishable from human content (to humans) and it will be hard to figure out what&#x27;s in your training set.

LLM Kessler Syndrome.

I often think this about Anki and spaced repetition. At the limiting case it has to be overwriting other memories, right?

Or in short words we have an upcoming AI collapse as AI output bleeds into the internet space where AI is collecting their inputs in the first place.<p>Actually, not scary as it forces everyone to look for solutions both the in-box kind and the out-of-box kind.

Just weigh your training data by the engagement it has obtained online. High quality output will have more engagement.

I wonder if human learning will be similarly impaired.

I asked Bing&#x27;s GPT4 to summarize the abstract and then to summarize the summary for 11 times. I was starting a new chat after each summary. I was adding &quot;Summarize the following text:&quot; before each summary. I had to use the Creative mode else Bing was searching instead of summarizing.<p>1st summary: The text discusses the impact of large language models (LLMs) on online text and image generation. It argues that LLMs can cause model collapse, a phenomenon where the original diversity of data is lost due to feedback loops between models and data sources. It shows that model collapse can affect various types of generative models and warns that it poses a threat to the quality and reliability of online content. It suggests that data from real human interactions will become more valuable to prevent model collapse.<p>2nd: The text warns that large language models (LLMs) can cause online content to lose its original diversity and become less trustworthy. It explains that LLMs can create feedback loops that reduce the variety of data they learn from. It suggests that real human data will be more important to avoid this problem.<p>3rd: The text argues that LLMs can make online content more homogeneous and unreliable by creating feedback loops that limit their data sources. It recommends using real human data to prevent this issue.<p>4th: The text warns that LLMs can produce biased and repetitive online content by relying on narrow data sets. It suggests using diverse and authentic human data to avoid this problem.<p>5th: The text advises using varied and real human data to prevent LLMs from creating online content that is unfair and redundant due to limited data sources.<p>6th: The text suggests using diverse and authentic data to avoid biased and repetitive online content by LLMs.<p>7th: The text argues that LLMs need better data to produce more varied and realistic online content.<p>8th: The text claims that language and literature models (LLMs) require more diverse and accurate data to generate online content that reflects the real world.<p>9th: The text argues that LLMs need better data to create realistic online content.<p>10th, Bing started to hallucinate, I double-checked that it didn&#x27;t search and the chat was new: The text claims that online content creators who use language models (LLMs) should have access to more reliable and diverse data sources to produce realistic and accurate content. It warns that LLMs can generate misleading or harmful content if they are trained on biased or low-quality data. It also suggests some ways to improve the data quality and availability for LLMs, such as using data verification tools, crowdsourcing data collection, and collaborating with other content creators.<p>11th: The text argues that online content creators need better and more diverse data sources to use language models (LLMs) effectively. It cautions that LLMs can produce harmful or inaccurate content if they rely on flawed data. It also recommends some methods to enhance the data quality and access for LLMs, such as verifying data, crowdsourcing data, and cooperating with other creators.