This thing where AI can improve itself seems to me in violation of the second law. I'm not a physicist by training merely an engineer but my argument is as follows:<p>- I think the reason humans are clever is because nature spent 6 billion years * millions of energetic lifetimes (that is, something on the order of <i>quettajoules</i> of energy) optimizing us to be clever.<p>- Life is a system, which does nothing more than optimize and pass on information. An organism is a thing which reproduces itself, well enough to pass its DNA (aka. information) along. In some sense, it is a gigantic heat engine which exploits the energy gradient to organize itself, in the manner of a dissipative structure [1]<p>- Think of how "AI" was invented: all of these geometric intuitions we have about deep learning, all of the cleverness we use, to imagine how backpropagation works and invent new thinking machines. All of the cleverness humanity has used to create the training dataset for these machines. This <i>cleverness</i>. It could not arise spontaneously, instead, it arose as a byproduct, from the long existence of a terawatt energy gradient. This captured energy was expended, to compress <i>information/energy</i> from the physical world, in a process which created highly organized structures (human brains) that are capable of being clever.<p>- The cleverness of human beings and the machines they make is, in fact, nothing more than the byproduct of an elaborate dissipative structure whose emergence and continued organization requires enormous amounts of physical energy: 1-2% of all solar radiation hitting earth (terawatts), times 3 billion years (existence of photosynthesis).<p>- If you look at it this way it's incredibly clear that the remarkable cleverness of these machines is nothing more than a bounded image, of the cleverness of human beings. We have a long way to go, before we are training artificial neural networks, with energy on the order of 10^30 joule [2]. Until then, we will not become capable of making machines that are cleverer than human beings.<p>- Perhaps we could make a machine that is cleverer than one single human. But we will never have an AI that is more clever than a collection of us, because the thing itself must be, in a 2nd law sense, less clever than us, for the simple reason that we have used our cleverness to create it.<p>- That is to say that there is no free lunch. A "superhuman" AI will not happen in 10, 100, or even 1,000 years, unless we find the vast amount of energy (10^30J) which will be required to train it. Humans will <i>always</i> be better and smarter. We have had 3 billion years of photosynthesis, this thing was trained in what, 120 days? A petajoule?<p>[1] <a href="https://pmc.ncbi.nlm.nih.gov/articles/PMC7712552/" rel="nofollow">https://pmc.ncbi.nlm.nih.gov/articles/PMC7712552/</a><p>[2] Where do we get 10^30J?<p>Total energy hitting earth in one year: 5.5×10^24 J<p>Fraction of that energy used by all plants: 0.05%<p>Time plants have been alive on earth: 3 billion years<p>You get to 8*10^30 if you multiply these numbers. Round down.