For everyone who considers the Chinese Room argument obviously wrong (like me), here's two versions that are much stronger and might still make you uneasy:<p>In Greg Egan's "Jewelhead" stories, every person gets a computer implanted in their brain at birth, which gradually learns to imitate the input-output behavior of the biological brain. At some point they switch to the jewel full time and throw away the biological brain, becoming immortal. That's seen as a fact of life and people don't question it much.<p>In one of Wei Dai's nightmare scenarios, we ask an AI to upload humans in an efficient way. Unfortunately, since humans can't introspect into the idea of "I'm conscious" very deeply, the resulting resource-optimized uploads just have a handful of hardcoded responses to questions about consciousness, and aren't in fact conscious. Nobody notices.<p>Of course, both cases are problematic only if you can "optimize" a human brain into something else, which would mimic the same input-output behavior without being conscious. The trouble is that we can't rule out that possibility today. Humans certainly have a lot of neural circuitry that's a side effect of something else. Some of it might get optimized out, the way a human in a sealed room can be optimized to nothing at all. To rule out a "Disneyland without children" scenario, wishful thinking isn't enough, we need to properly figure out consciousness and qualia.