The problem I have with this is that yes, I understand the possibilty of peverse incentives, but let's look at our real world experience. When we play a game with a smart animal or a child and they figure out how to cheat in order to get their reward we don't suddenly become lobotomized. We see their cheating, and generally find it amusing or cute. In the same way I think it's one thing to say that an AI might cheat to acheive a goal, but it's a completely different thing to say it'll be competent. Like all of this is really interesting, but it kind of assumes a lot.<p>Let's say that this machine breaks out and decides to take over the world to acheive whatever stupid task we've given it (I'm yet to see an AI ethics paper that doesn't exclusively deal in stupid tasks), do we really think that the AI is going to figure out it's going to need to ensure it stabilizes the power grid before it starts exponentially drawing power to collect buttons or whatever it is?<p>What is much more likely from our experience is that a basic AGI is going to do something really dumb. And then when we reboot it it'll do something dumb 1000x more times. And maybe eventually it'll do something almost as smart as us. This is what we term childhood.<p>I think a core part of these ethical papers is that you suppose an AI is generally intelligent, and that it is <i>much</i> more intelligent than anything plausible and never needs to learn from experience. Let me tell you, I'm pretty generally intelligent and I've not taken over the world more times than I can count.