This wasn't obvious to me a while ago, but I've come around to it. To me the important thing is that it's not only compression, it's <i>lossy</i> compression.<p>Models (any kind) are basically an imperfect approximation of some system, and that definition is precisely what's identified here.<p>You can demonstrate it by assuming some sequential process that gets an "AI" like an LLM to generate as much content as it can[1], then train a new generation model on this content. Then use the new model to generate as much content as <i>it</i> can, train a third generation model, and so on.<p>LLM->generate->LLM'->generate'->LLM''->generate''->...->LLM'...'<p>Since each generation may not produce content with <i>every</i> possible token it has been trained on, it stands to probability that eventually some tokens, especially low probability tokens, will simply be left out of the generated content. The next model will therefore <i>lose</i> that token and it will never be part of the content <i>it</i> generates, and so on.<p>After enough generations of this process, you eventually end up with a model with a single, or no, tokens it can use to generate text.<p>The problem is that the models are trained on <i>so</i> much information that they're effectively models of the textual content of our civilization. If we hand off the reigns to these systems, they may work well for a while, and may even seem to be produce novel ideas (but which are secretly just old ideas in combinations we haven't seen before), but they'll also produce content which will inevitably be used to train later models. As the content these models produces starts to dominate the ratio of human-text:model-text, there will be less of a reason to preserve the human text and we'll end up with the scenario above.<p>Things may work well for a while, perhaps a <i>long</i> time, but even if we don't end up with models trained on one token, we will end up with fewer and fewer ideas-as-text represented within them. Civilization will stagnate, the singularity will not commence, and things will slowly regress.<p>Of course this all presumes research stops, and somehow LLMs become religion and we turn society over to them. Which of course isn't what's going to happen. Is it?