In his recent talk Ilya S. said that the data running out is a fundamental constraint on the scaling laws.
He said "we have but one internet".<p>But I don't understand: there is so much data in the real world beyond the internet.
Webcams. Microphones. Cars. Robots...
Everything can collect multimodal data and more importantly (for robots) even get feedback loops from reality.<p>So isn't data functionally infinite? And the only thing standing in the way is the number of sensors and open datastreams and datasets.<p>Please help me understand
There's a ton of recent work on data curation / synthetic data generation that shows that smaller high quality datasets go a lot further than scaling up on noisy web data.<p>The scaling law plots are log scale so to get more juice with naive scaling we'd need to invest exponentially more resources, and we're at a point where the juice is not worth the squeeze, so people will shift to moving the curve down with new architectures, better curated datasets and test time compute / RL.<p>See:<p>- FineWeb: <a href="https://arxiv.org/abs/2406.17557" rel="nofollow">https://arxiv.org/abs/2406.17557</a><p>- Phi-4: <a href="https://arxiv.org/abs/2412.08905" rel="nofollow">https://arxiv.org/abs/2412.08905</a><p>- DataComp: <a href="https://arxiv.org/abs/2406.11794" rel="nofollow">https://arxiv.org/abs/2406.11794</a>
If you show an LLM one webcam feed to train on, that's useful. Two is even more useful. But there are diminishing returns. "Useful training data" is limited.<p>Humans get to PhD level with barely a drop of training data compared to what LLMs are trained on.<p>If there were infinite useful data, then scaling AI on data would make sense. Since there isn't, the way forward is getting more efficient at using the data we have.
Think about it from his perspective.<p>Data from the internet can be chunked, sorted, easily processed, and has a relatively high signal-to-noise ratio. Data from a webcam or a microphone -- if even legal to access in the first place -- would be a mess. Imagine chunking and processing 5TB of that sort of data. Seems to me that the effort would far outweigh the reward.<p>Robots are a different problem entirely. It's darkly amusing that simple problems of motion through space are more complex to replicate than painting the simulacra of a masterpiece, or acing the medical licensing exam. We'll probably have AGI before we can mimic the movement of the simple housefly.
There are lots of problems where someone has to run experiments to generate data. If the most optimized possible process to perform the experiment is expensive and takes time to generate 1 data point, then all you can do is wait till more data is produced before a solution is found. Think drug discovery.
Complex systems studies is wisdom. We know how communication on internet behaves. Conway's Law hits hard and the processes of life are not dumb.<p>Access to physical reality is important when negotiating with the beings that can form under this constraint. People have apparently known this instinctively for a very long time and they are not going to give in to the demands of the AI industry.<p>It's a great mistake to humanize everything in your consciousness.