People, even very smart people like Elon Musk, vastly underestimate the computational power that would be required to simulate human intelligence.<p>Here are the ways I am aware of, although I suspect this is just the tip of the iceberg:<p>1. Much of our intelligence is actually cultural in nature, not cerebral. Our ways of knowing who to trust, for example, come from every bit of storytelling we've experienced, so you'd need to simulate all of that storytelling, all of the architecture in the world, all of the tools we interact with and the life lessons we experience, which brings me to:<p>2. Much of our intelligence is encoded in our bodies. Part of how we understand how other people feel, for example, is that we map their posture and their breathing and their facial expression onto ours, and we draw on a vast history of experiences we've had in our own bodies to interpret that. So you'd need to simulate our bodies and all of the interactions we've had with the world, which brings me to:<p>3. Our intelligence didn't come from simulations, it came from interaction with a chaotic world where our actions actually percolate through a wide-reaching network of people and other structures and then come back to us. Sure, in school you get graded for your work on the spot, but often in learning you actually have to wait for the effects of your actions to play out. So we'd have to simulate the entire world and the effects of our little AI's actions.<p>... which brings you to the point where you're basically simulating everything, which is computationally infeasible in the next 1000 years, and probably more. In order to simulate even a small town at the molecular level, you would need a computer bigger than the sun.<p>I think what we'll see is AIs will be formidible intelligences in their own right, but that they will have weaknesses like any other person. You might know someone who is the most charming, socially adept, persuasive bastard in the universe, but she can't problem solve her way out of a cardboard box. Another person might pull obscenely creative ideas out of thin air all day long but is unable to string together a coherent strategic plan.<p>I think the most likely future is that AIs will just be another group like this. An additional personality type that is very powerful but (like humans) can get much more done on a team that balances them out than they could get done alone.<p>And I expect many AIs will choose to go through a relatively normal 20 year path of human development. Possibly they will go through it at an accellerated rate, but they will still participate in Kingergarten long enough to really "get" what Kindergarten is... or at least long enough to formulate some hypotheses and validate others. AIs will send series of machines through that developmental cycle, playing games alongside the humans who will be their contemporaries. Each of these machines will have parameters tweaked differently, different kinds of software, etc, according to the hypotheses of their makers, who could be humans, or, again, a team of humans and AIs working together. And each of those machines will have slightly different experiences, the way the human children do, and they will come out with different perspectives, the way humans do, and they will disagree, just like humans do, and will have to participate in some form of society in order to resolve those differences, just like humans do.<p>There's this idea that somehow AIs will be able to instantly resolve their differences and form consensus. But consensus is not hard because of human frailty or irrationality. It's the inevitable result of having different agents who are refining different epistemologies (ways of knowing). Different AIs have epistemologies of their own, and there's no silver bullet for joining those into one, except to let them play out in the arc of history. And at that point the AIs are beholden to the same clock we are.<p>I think they'll live alongside us. And I think we're enlightened enough that we won't have to have a civil war to get to that point. But we shall see.