Key takeaways:<p>- The training compute of notable AI models is doubling roughly every six months.<p>- Training compute costs are doubling every nine months for the largest AI models.<p>- Training compute has scaled up faster for language than vision.<p>- The size of datasets used to train language models doubles approximately every eight months.<p>- The length of time spent training notable models is growing.<p>- The power required to train frontier AI models is doubling annually.<p>- Leading AI companies have hundreds of thousands of cutting-edge AI chips.<p>Those are the conclusions for each section with some explanation for the data in each section. The implications seem to be that production of these LLMs is getting ever more costly, especially in terms of time and energy, even as training and algorithms become more efficient and more effective.<p>Is this all supply-side with the assumption of demand, or is the demand curve already rising to match?
Damn, by the year 2028 (median, predicted based on current trends) all human generated content would be used to train the models. Baffles me, considering how cold the AI winter really was, even a few years back.