> The bottom line is that automation of the computer codes and the algorithms are designed by people, which is what they are actually being designed to protect against<p>If this can happen in a multi-million dollar airplane that has 3 computers, it is no far stretch to see it happening with self-driving vehicles that only have 1 - and much less R&D. The upside being you're not 35K feet in the air.<p>There is a phrase I like to murmur to myself that I think is applicable here: "Apple knows best". As of late, Apple has made numerous technology and software decisions that I find quite disagreeable. And many of these have, in fact, significant implications for myself, consumers, and their products overall. But "Apple knows best!" In their infinite wisdom, they call all the shots, good and bad. And if it's a bad one, which to me many are... you suffer with no recourse, because there is ONLY their way.<p>I feel this relevant because one of the main issues the pilots faced is that the computers are <i>above</i> them in command. They are authoritative - and so don't reveal things to their underlings. In a computer system that is meant to have no single point of failure, they neglected to realize that the computer system itself is a single point of failure. This means the designers of the airplane, purposefully removed control from pilots and gave them no recourse or alternatives.<p>These are the lessons we need to learn from. While automation of flight has improved safety and reduced human error, it does not allow for humans to reduce automation error. In symbology, the two can work together to improve safety even more, as this article makes clear. With better communication from the computers to pilots, and better ability for pilots to seize control from systems that are misbehaving, there is a greater chance for success.<p>But I think this solution is in contrast of where the industry wants to go: to remove pilots entirely. And for the bottom line of airlines, that's pretty dreamy. But at what cost to potential lives saved? If QF72 was solely automated, this would have been a very different story.