Not sure I follow, how would these scenarios be any different if a human was making the decision? Some humans will act with self interest, some with a utilitarianism, and some just simply will freeze. I'm not so sure it would make much a difference how the car chooses.<p>So, philosophical question aside, the concern for autonomous vehicle fleets having backdoor and being abused is more concerning..
I love the tee-up: "Should autonomous vehicles be programmed to choose who they kill when they crash?!"<p>...because the answer provided is approximately, "that's a stupid question."<p>It's like guardian didn't even read Doctorow's editorial before publishing.
The main assumption made by Cory Doctorow is that the autonomous cars will be fully automated and under the control of somebody other than the driver.
That scenario - with a Uber-like owner of a point-to-point fleet of self-driving cars - is indeed a bit concerning, considering that you, the passenger, give up control of the vehicle and you have to trust the machine with your life in order to get you from point A to point B.<p>But this is exactly why I think that the consumer version of the self-driving car will always have a manual drive setting that will completely disengage the auto-pilot.<p>The insurance companies will never insure at an acceptable cost point a car that cannot yield control to its passenger that is the designated driver.<p>Also, passenger will want the reassurance that at any point they can take over the car and switch it off the auto pilot.<p>iPhones and any other mobile devices are a bad analogy with a self-driving car. The more apt comparison is with the current modern airplanes: even if these planes can drive themselves, the pilot is able at any time to take over.<p>This option is needed even more on the roads, where you don't have the dedicated flight path a plane has in the air.
Aren't these two separate questions?<p>Should autonomous vehicles be programmed to kill their owners in some circumstances?<p>and<p>Should owners of autonomous vehicles be legally allowed to change the vehicle's code?<p>Both have their own separate legal and moral quagmires.
Cory is of course completely and utterly wrong here on a technical level. Indeed, what he's saying is the exact opposite of the truth. Locks that prevent you from modifying the code running on your hardware don't make it easier for an attacker to break in, it's the exact opposite - any feature that lets you install modified code can also be used by an attacker to install malicious code that will steer your car into a bridge on command. This is why some experts recommend the use of locked-down devices like iPhones if your hardware might be targeted by an attcaker.<p>Likewise, there is no requirement that "A digital lock creates a zone in a computer’s programmer that even its owner can’t enter. For it to work, the lock’s associated files must be invisible to the owner." You need that for DRM because the purpose of DRM is to protect access to files, but a properly-locked-down system will not let you modify the code even if you can look at every single piece of code and data on the system.
Another reason why self-driving cars should have manual controls:<p>There is a road near my house that is closed by the neighboring school at random. Google treats this road as a normal road when choosing routes via Maps.<p>If a self-driving car drives up to the gate, that's at least a lot of wasted time, and if the car can't detect the chain-link gate...
What I wonder about most is how we are going to do the certification of the code. Given that the code is probably far too difficult to analyze manually, I assume that certification will mostly be a matter of letting the car drive for X miles and seeing if it doesn't crash.<p>The problem that arises then is that after every tiny update, do we have to go through the complete certification process again? It is not possible to simply record and replay sensor information, since every change in behavior may result in a change in the environment, and you cannot predict the environment.
What I see as inevitable is a transition to software engineering being a licensed profession. And additionally the application of formal methods of system validation.
Title changed from "Cory Doctorow on Software Security and the Internet of Things". Submitters: please use the original title unless it is misleading or linkbait, as the site guidelines ask.<p><a href="https://news.ycombinator.com/newsguidelines.html" rel="nofollow">https://news.ycombinator.com/newsguidelines.html</a>