Some of my thoughts after reading this article:<p>Everyone seems to believe that achieving artificial general intelligence is inevitable. I'd argue that it's only inevitable if humanity survives long enough to make it happen. I'm not a pessimist, but the next 300-400 years will be the most difficult humanity has ever faced. In addition to climate change, population expansion, nuclear weapon proliferation, and naturally increasing inequality, humanity will face many more threats that haven't yet been perceived.<p>I believe building strong intelligence will optimistically take 3-4 centuries. To complete a "system that could successfully perform any intellectual task that a human being can," it is first necessary to understand what defines the human intellect and conscience. Although there is much ethical debate on what defines a conscience, scientifically it is a product of experience, and can be emulated if three requirements are met:<p>- The system can accept every input the human body can.
- It can process every combination of stimuli in the same way a human would.
- It can to the stimuli in every way a human would.<p>For these requirements to be met by software, we must either:<p>- Aquire a nearly full knowledge of the human brain (and probably body)'s information-processing mechanisms, and figure out how to implement this in software.
- Build enough processing power to completely simulate the human brain/body, atom by atom.<p>Both will not be feasible for an extraordinary amount of time, and it's probably better to spend our time worrying about the current existential threats to humanity.<p>A much more relevant ethical problem for humanity is that of eugenics. Unlike AI, recent advances like CRISPR/CAS-9 make it viable now, through modification of sperm-generating stem cells (not embryos), and like AI, it offers world-changing benefits (the eradication of diseases, lengthened life spans, increased knowledge and strength, etc), while also providing the keys for modern humanity's destruction (designer babies, lack of diversity, separation of humans into casts, etc).<p>Perhaps increases in intelligence driven by eugenics will even cause computer systems and AI to become obsolete.