The threat isn't that robots will kill you, but that they will take your livilihood. If you are a worker in a capitalist society, you should be worried. The analogy is the internal combustion engine vs the horse. What happens when AI advances to the point of matching your general intelligence and learning capacity? If you work in IT, poof, firm management outsources your job to a computer. If the AI is attached to a robot that further matches your dexterity, poof there go most other jobs as well.
If you're worried that a sentient robot is going to kill you, why are you not first worried about a non-sentient robot being able to kill you? If robots are in a position where they can kill, sentient or not, <i>humans</i> will exploit them to do this far before an AI is conscious enough to do so. There is plenty of volition for this from humans, e.g. terrorism.
I think the "high profile people" are not afraid of the robotic entity being nativity good or evil rather the affect the AI device has on humanity being detrimental to overall human advancement. For example, look at how smart phones have caused people to only be able to recall a handful of phone numbers.