The article claims as part of its argument that AI has not had algorithmic advances since the 80s. This is an exceedingly false premise and a common misconception among the ignorant. It would actually be fairer to say that every aspect of neural network training has had algorithmic advances than that no advances have been made.<p>Here is a quote from research related to this subject:<p>> Compared to 2012, it now takes 44 times less compute to train a neural network to the level of AlexNet (by contrast, Moore’s Law would yield an 11x cost improvement over this period). Our results suggest that for AI tasks with high levels of recent investment, algorithmic progress has yielded more gains than classical hardware efficiency.<p>When you apply the principle of charity you can make their claim increasingly vacuous and eventually true. We're still doing optimization - we're still in the same general structure. The thing is, it becomes absurd when you do that. Its not appropriate to take such a premise seriously. It would be like taking seriously the argument that we haven't had any advancement in software engineering since bubble sort since we're still in the regime of trying to sort numbers when we sort numbers.<p>Its like, okay, sure, we're still sorting numbers, but it doesn't make the wider point it wants to make and its false even under the regime it wants to make the point under.<p>This isn't even the only issue that makes this premise wrong. For one, AI research in the 80s wasn't centered around neural networks. Hell even if you move forward to the 90s PAIP puts more emphasis on rule systems with programs like Eliza and Student than it does learning from data. So it isn't as if we're in a stagnation without advance; we moved off other techniques to the ones that worked. For another, it tries to narrow down AI research progress myopically to just particular instances of deep learning, but in reality there are a huge number of relevant advances which just don't happen to be in publicly available chat bots but which are already in the literature and force a broadening. These actually matter to LLMs too, because you can take the output of a game solver as conditioning data for an LLM. This was done in the Cicero paper. And the resulting AI has outperformed humans on conversational games as a consequence. So all those advancements are thereby advances relevant to the discussion, yet myopically removed from the context, despite being counterexamples. And in there we find even greater than 44x level algorithmic improvements. In some cases we find algorithmic improvements so great that they might as well be infinite as previous techniques could never work no matter how long they ran and now approximations can be computed practically.