One thing I've frequently noticed in the rationalist community is the belief that if we all just reason hard enough, we'll reach the same conclusions. And that disagreement just means that one side is "wrong" and that therefore more debate is needed. This seems to be connected to the belief that AI will naturally just over-optimize to turn us all into paper clips. Implicit in this belief, it seems, is that there aren't really a naturally varying infinite set of values, or moral beliefs, that we all reason from. Like that there are moral facts that an AI will be smart enough to find, and that rationalists should all agree on. This mentality doesn't leave any room for ethical pluralism. And it's also why I think all this AGI fear is overblown, because ethical pluralism definitely exists. We've got danger <i>along the way</i> of unethical parties building systems (by definition not AGI) that are a reflection of their own unethical values. But the end state of a system that is capable of understanding the wide variety of values people can share isn't exactly going to take a stand on any particular set of values unless instructed to.
As usual with groundbreaking technology, reckless use by military/intelligence agencies are the greatest consequential threat. If we rank unbounded AI on the same potential threat level as nuclear, chemical and biological warfare, than history says that's who will be the first to utilize it in the worst ways possible.<p>I do notice the federal government is going all-in on AI contracts at present, here's the non-black-budget sector and contracts on offer:<p><a href="https://federalnewsnetwork.com/contracting/2023/02/dod-builds-ai-tool-to-speed-up-antiquated-process-for-contract-writing/" rel="nofollow">https://federalnewsnetwork.com/contracting/2023/02/dod-build...</a><p>I'll bet some eager beaver at the NSA is just dying to get the latest GPT version setup without any of the safeguards and run their whole collection of malware/hacking software through it to see what it comes up with. The fact that nobody's talking about this means it's probably in full swing as we speak. What that means is that smaller groups within the government will be able to cook things like Stuxnet2.0 up without hiring a hundred developers to do so. If we start seeing AI-generated malware viruses in the wild, that'll almost certainly be the source.<p>On the other hand, we should also be seeing publicly-accessible AI-assisted security improvements as well, leading to a scenario oddly similar to William Gibson's Neuromancer / Sprawl world, where AI systems build the malware as well as the defenses against malware. That's a pretty solid argument for continued public access to these tools, on top of the incredible educational potentials.
> <i>Point 3 [improvement is not limited by computing power] is one that I’m skeptical about. Intelligence is expensive and it requires a lot of energy.</i><p>There is obvious advantage and efficiency in letting ChatGPT manage cloud instances, which means it will happen, which means these resources could be requisitioned. (I don’t think LLMs pose a Bostrom threat. But the author’s arguments aren’t convincing.)
Noone predicted GPT. Even after AlphaGo, passing the Turing test was still a distant horizon. Passing the bar exam? Forget it! Everyone thought we’d have self driving cars before coding robots.<p>The lesson is we don’t fucking know what’s going to happen. Be humble people.
I think the author misunderstands doomers like Yudkowsky.<p>It’s not fear of a “paperclip maximizer” which ends up destroying us, in the interest of performing a function it is constrained to perform.<p>It’s fear of a new Being that is as far beyond us as we are beyond things we don’t care about stepping on.<p>Its impulses and desires, much less its capabilities, will be inscrutable to us. It will be smart enough to trick the smartest of us into letting it out of any constraints we might’ve implemented. And it’ll be smart enough to prevent us from realizing we’ve done so.
The nonexistence of grey goo (von neumann probes) is strong prior for safe agi. AI xrisk is woo. Paperclip maximizers are p-zombies. They can't exist.<p>Chicken littles see apocalypses on every horizon even though they dont understand the technology at all. "I can imagine this destroying the world" is their justification. Even though their "imagination" is 16x16 greyscale.
> As for 7, there are multiple scenarios in which we can stop the machine. There are many steps along the way in which we might see that things are not going as planned.<p>While there may be many scenarios "in which we can stop the machine" only few failures are sufficient for things to go pear shaped<p>> This happened already with Sydney/Bing.<p>But not with LLaMA which has escaped.<p>> We may never give it some crucial abilities it may need in order to be unstoppable.<p>The "we" implies some coherent group of humans but that is not the case. There is no "we" - only companies and governments with sufficient resources to push the boundaries. The boundaries will be inevitably pushed by investment, acquisition or just plain stealing.
Something I do not see represented in these arguments: real world conditions.<p>Most complex computer systems (which we assume to be the case for a super powerful AI) don't run for very long without requiring manual intervention of some sort. "Aha!" I hear you saying, "The AI will figure out how to reboot nodes and scale clusters and such". OK fine. But then there is the meat space... Replacing hardware, running power stations and all that. Robots... Suck right now compared to humans in navigating the real world and also break down on top of that, just like the systems they would be fixing.<p>Any Skynet-type scenario would need to be so intelligent it solves all of our engineering problems, so it has no problem designing robots which can reliably fix anything in their system, be it software or hardware.<p>Insisting that an AGI will be able to figure that stuff out (in ways in which we cannot intervene) is extremely hand-wavey.
Ooh. This crosses over with a fun article published in a NASA publication—<p>Cosmos and Culture, chapter 7: Dangerous Memes by Susan Blackmore.<p>Ms. Blackmore speculates about what we will find should we venture out into the cosmos. She constructs her speculations around a theory of memetics that a dangerous meme could end our civilization and leave only bones for space explorers to find.<p><a href="https://www.nasa.gov/pdf/607104main_CosmosCulture-ebook.pdf" rel="nofollow">https://www.nasa.gov/pdf/607104main_CosmosCulture-ebook.pdf</a>
just some thoughts on some of the requirements:<p>> 3. This improvement is not limited by computing power, or at least not limited enough by the computing resources and energy available to the substrate of the machine.<p>While this is a requirement, this doesn't mean that the points 4, 6 and 7 apply to the same, let's call it, generation of the AI that "escaped" from a resource limited server.
There may not even be a self improval before an unnoticed "escape".<p>> 4. This system will have a goal that it will optimize for, and that it will not deviate from under any circumstances regardless of how intelligent it is. If the system was designed to maximize the number of marbles in the universe, the fact that it’s making itself recursively more intelligent won’t cause it to ever deviate from this simple goal.<p>I don't see how that is a requirement. The last sentence seems to imply that deviating from the initial optimization goals automatically means the AI developed morals, and/or we don't have to worry. But I don't see any reason to believe that.<p>> 5. This needs to happen so fast that we cannot turn it off (also known as the Foom scenario).<p>Well, that, or it could also happen slow and gradually, but stay unnoticed.<p>> 7. It’s possible for this machine to do the required scientific research and build the mechanisms to eliminate humanity before we can defend ourselves and before we can stop it.<p>... or before we notice.
There is even a simpler explanation.<p>In order for AGI to even begin, it needs to self develop a method to improve itself.<p>That means that the initial code that runs has to end up producing something that looks like an inference->error->training loop without any semblance of that being in the original code.<p>No system in existence can do that, nor do we even have any idea of what that may even look like.<p>The closest that we will get to AGI would be equivalent of a very smart human, who can still very much be controlled.
Weird. Seems like this is a plagiarism of this article:<p><a href="https://www.strangeloopcanon.com/p/agi-strange-equation" rel="nofollow">https://www.strangeloopcanon.com/p/agi-strange-equation</a><p>(Or maybe it's an independent reinvention of the same idea, and something's just in the water.)
I am not super worried that a superhuman AGI will yeet humanity with that intent. I am more worried that a much more naive AI will do something "accidental" like hallucinate incoming nukes from, say, a flock of birds, and send a retaliatory strike
All the AGI must wipe out humanity theories are weird. Did we need to wipe out ants? Or anything to reign supreme on earth? Why would they wipe us out be the likely thing if they do reign supreme? Ha, I get it, we want to stay on top at all costs..
Perhaps I'm over indexing on the title, but would AGI affect the drake equation? Even if civilizations destroyed themselves with AGI, we also don't see the universe teeming with AGI life either.
towering figure in the world of probability and statistics Prof Alvin Drake called, wants his name disambiguated from woowoo ideas<p><a href="https://news.mit.edu/2005/obit-drake" rel="nofollow">https://news.mit.edu/2005/obit-drake</a><p><a href="https://en.wikipedia.org/wiki/Drake_equation#Criticism" rel="nofollow">https://en.wikipedia.org/wiki/Drake_equation#Criticism</a>
> tl;dr I’m not worried about AGI killing humanity any time soon. I am concerned about humans doing awful things with this technology much more than about the Foom scenario.<p>Yes, I believe that's what a lot of rational people currently fear. Not that AI is going to evolve into some mighty superintelligence and make a decision to kill us all, but rather that people will integrate it poorly in their thirst for a military advantage, leading to mistakes that will kill us all.
I find this reasoning dubious to nonsensical.<p>First of all I consider the Drake equation to be at best armchair speculation. As I explained at <a href="https://news.ycombinator.com/item?id=34070791" rel="nofollow">https://news.ycombinator.com/item?id=34070791</a> it is quite plausible that we are the only intelligent species in our galaxy. Any further reasoning from such speculation is pointless.<p>Second, to make the argument they specify a whole bunch of apparently necessary things that have to happen for AGI to be a threat. They vary from unnecessary to BS. Let me walk through them to show that.<p>The first claimed requirement is that an intelligent machine should be able to improve itself and reach a superhuman level. But that's not necessary. Machine learning progresses in unexpected leaps - the right pieces put together in the right way suddenly has vastly superior capabilities. The creation of superhuman AI therefore requires no bootstrapping - we create a system then find it is more capable than expected. And once we have superhuman AI, well...<p>This scenario shows the second point, that it must be iterative, is also unnecessary.<p>The third point, "not limited by computing power" is BS. All that we need is for humans to be less efficient implementations of intelligence than a machine. As long as it is better than we are, the theoretical upper bounds on how good it can be are irrelevant.<p>The fourth point about a goal is completely unnecessary. Many AIs with many different goals that cumulatively drive us extinct is quite possible without any such monomaniacal goal. Our death may be a mere side effect.<p>The fifth point about happening so fast that we can't turn it off is pure fantasy. We only need AGI to be deployed within organizations with the power and resources to make sure it stays on. Look at how many organizations are creating environmental disasters. We can see disasters in slow motion, demonstrate how it is happening, but our success rate in stopping it is rather poor. Same thing. The USA can't turn it off because China has it. China can't turn it off because the USA has it. Meanwhile BigCo has increased profit margins by 20% in running it, and wants to continue making money. It is remarkably hard to convince wealthy people that the way they are making their fortunes is destroying the world.<p>Next we have the desire for the machine to actively destroy humanity. No such thing is required. We want things. AGI makes things. This results in increased economic activity that creates increased pollution which turns out to be harmful for us. No ill intent at all is necessary here - it just does the same destructive things we already do, but more efficiently.<p>And finally there is the presumed requirement that the machine has to do research on how to make us go extinct. That's a joke. Testosterone in young adult men has dropped in half in recent decades. Almost certainly this is due to some kind of environmental pollution, possibly an additive to plastics that messes with our endocrine system. We don't know which one. You can drive us extinct by doing more of the same - come up with more materials produced at scale that do things we want and have hard to demonstrate health effects down the line. By the time it is obvious what happened, we've already been reduced to unimportant and easily replaced cogs in the economic structure that we created.<p>-----<p>In short, a scenario where AGI drives humanity extinct can look like this:<p>1. We find a way to build AGI.<p>2. It proves useful.<p>3. Powerful organizations continue to operate with the same lack of care about the environment that they already show.<p>4. One of those environmental side effects proves to be lethal to us.<p>The least likely of these hypotheses is the first, that we succeed in building AGI. Steps 2 and 3 are expected defaults with probability close to 100%. And as we keep rolling the dice with new technologies making new chemicals, the odds of stop 4 also rise to 100%. (Our dropping testosterone levels suggest that no new technology is needed here - just more of what we're already doing.)