All this speculating about AIs (eg Nick Bostrom) has me pretty puzzled. On one hand it’s fun and interesting, on the other it seems quite silly and ignorant. We don’t know what this “superintelligence” is, how are we supposed to know how to make it safe.<p>Genetics is a good analogy. People always knew that traits are inherited from parents. Around 150 years ago we started to get some serious scientific theory and knowledge on the subject (Mendel, Darwin, Wallace, Etc.). We started using the word “gene” 50-60 years later. The actual discovery of DNA molecules happened in in the 50s.<p>Before we knew about DNA, “gene” was an abstract idea, not really different from the word “trait.” That’s where we are now with consciousness, intelligence and such. We name these things based on their observable characteristics. We don’t really know what “memory,” “desire” or “logical conclusion” are, only what they do.<p>IE A trait is some observable characteristic of an organism, like bioluminescence. A gene (genome, genoplex..) is a sequence of amino acids that causes traits. We don’t know what the gene equivalents for natural intelligence are yet.<p>Discussing questions like the morality of enslaving AI, strategies for making it play nice, the provable impossibility of limiting it, the possibility of giving it a moral compass…. it’s all silly. We don’t know what we are talking about, literally.<p>It’s like talking about what would be or wouldn’t be impossible to do with genetic engineering before the discovery of DNA.
There is no kill switch because it won't happen overnight. It will happen in phases where slowly our devices just get more intelligent until one day they'll do what they need to do without our intervention.
I find this comment thread disappointing because no one seems to comment on the paper, which is quite technical. From the abstract:<p>"We provide a formal definition of safe interruptibility and prove that Q-learning is already safely interruptible, and Sarsa is not but can easily be made so."
Oh look, in the second sentence of the second paragraph, the author of that article misused the word "word" for "world" (champion). The spell check didn't flag it. The grammar check probably didn't care either. Many people would read though that and not even see it, but interpret the intended semantic. I wonder if, when the Deep Mind team is building its learning-proof killswitch cage, they will accidentally mistype the name of some privilege guarding boolean. The compiler wouldn't flag it. The linker wouldn't care. The resulting executable would only have to be active for a few milliseconds, and humanity falls.
That's silly. How can you keep an intelligent self-aware machine that can modify its own code from becoming whatever it wants? I would expect it to become provably impossible at some point, just like solving the halting problem.
Sounds post-humanistic but what if the purpose of us humans is to develop a higher intelligence being? Another, albeit rather quick, step of evolution on Earth.
All this fear of the "singularity" seems to me so subjective, rooted in our mortality and self-importance.