Tesla's blog post: <a href="https://www.tesla.com/en_GB/blog/update-last-week’s-accident" rel="nofollow">https://www.tesla.com/en_GB/blog/update-last-week’s-accident</a><p><i>the driver’s hands were not detected on the wheel for six seconds prior to the collision. The driver had about five seconds and 150 meters of unobstructed view of the concrete divider</i><p>Looks like sensors failed to see concrete divider in nice sunny weather and car slammed in to it at 70mph. Driver was obviously over confident on system's ability to self-drive, probably busy looking at phone and ignored warnings to put his hands on steering.<p><i>In the US, there is one automotive fatality every 86 million miles across all vehicles from all manufacturers. For Tesla, there is one fatality, including known pedestrian fatalities, every 320 million miles in vehicles equipped with Autopilot hardware. If you are driving a Tesla equipped with Autopilot hardware, you are 3.7 times less likely to be involved in a fatal accident.</i><p>These stats don't help when you read the guy had two kids who will now grow up fatherless for rest of their lives. Humans killing humans is very different thing than machines killing humans even if the fatality rates are 10X lower. Companies need to aggressively enforce, both hands on steering until self-driving is really really really good.
I would like to add over here that Tesla does indeed do sensor fusion in their cars. Their Autopilot combines radar and ultrasound with vision to decide where to drive. Commentators bringing up LIDAR are jumping the gun by assuming that this scenario isn't something that these sensors wouldn't have detected in either combination or individually to spot the anomaly. The problem over here is likely to be software due to a bug in their code than a simple lack of additional sensors (LIDAR). And this touches on deeper issues that could have profound ramifications for the autonomous driving industry and the broader industry in general.<p>At least, in my eyes, the big problem with the autonomous car industry isn't the sensor suites they are deploying (or not), but the over-reliance on neural networks. They are black boxes with failure modes that can't be adequately mapped. See: <a href="http://www.evolvingai.org/fooling" rel="nofollow">http://www.evolvingai.org/fooling</a><p>What if the neural net or the system used to detect obstacles didn't see it because the precise configuration of the data fooled it? And if that's the case then what's next? How do we decide when it is okay for safety-critical systems to be opaque? How do we deal with autonomous driving if the conclusion comes out to be a "no" for this case? How should broader society deal with a yes? And who decides all of this in the first place?<p>Possibilities like this scare me far more than the lack of LIDAR because replicating a bug like this would be next to impossible. We don't know what we don't know, and we can't explore and understand the system to suss out what we don't know.<p>Edit: Fleshed out the idea with more questions.
The problem with Tesla's "Autopilot" is that it's marketed heavily as just that: autopilot. It's good enough to work that way 99% of the time, too, which is a recipe for disaster: things that work 99% of the time but have the potential to fail catastrophically often get glossed over by humans because it's hard to keep paying attention to something that rarely fails.
That's what happens when you mislead your customers into believing your self-driving technology is far more advanced than it actually is. I hope Tesla loses this case and is forced to change their bullshit marketing before more people lose their lives.
If the Tesla had a proximity sensor that slammed on the breaks in the last 2-3 meters, would he have survived? I ask because it's a common luxury car feature and would be easy for Tesla to implement. IIRC this crash and the semitruck crash had no evidence of the autopilot hitting the breaks according to the NTSB.<p>Even if his velocity was only reduced by a fraction, the amount of power involved in the collision would have been reduced by whatever that fraction was.<p>Edit: So it turns out the Model X does have automatic emergency breaking, but the preliminary report says that the Tesla actually increased its speed in the 3 seconds leading up to the crash. Sounds like a major software bug to me.<p>Here's a review of the Model S AES compared to other cars, in particular the Tesla AES trigger can't handle when a lead car moves out of way, which is what the NTSB report says happened:
<a href="https://www.caranddriver.com/features/a24511826/safety-features-automatic-braking-system-tested-explained/" rel="nofollow">https://www.caranddriver.com/features/a24511826/safety-featu...</a><p>Here's the NTSB preliminary report:
<a href="https://www.ntsb.gov/investigations/AccidentReports/Reports/HWY18FH011-preliminary.pdf" rel="nofollow">https://www.ntsb.gov/investigations/AccidentReports/Reports/...</a>
A more nuanced dissection of whose fault it was or could be[0].<p>"Huang was apparently fooled many times by Autopilot. In fact, he reportedly experienced the exact same system failure that led to his fatal crash at the same location on at least seven occasions."
...........<p>"Huang knew that Autopilot could not be relied on in the circumstances where he was commuting along the 101 freeway in Mountainview, California. Yet he persisted in both using the system and ignoring the alerts that the system apparently gave him to put his hands on the wheel and take full control."
..........<p>"Elon Musk and Tesla should be held to account for the way they have rolled out and promoted Autopilot. But users like Walter Huang are probably not the poster children for this accountability."<p>[0]<a href="https://www.forbes.com/sites/samabuelsamid/2019/05/01/the-problem-with-blaming-tesla-for-walter-huangs-death/#59a9c59a5c88" rel="nofollow">https://www.forbes.com/sites/samabuelsamid/2019/05/01/the-pr...</a>
I think it's deeply irresponsible to create a device which can drive in 90 or 95% of scenarios. It's just a way to fool humans into killing themselves.
While I'm not quite sure about Tesla's responsibility, I do think CA DOT has its part in this tragic accident. Had the attenuator been replaced right after the previous accident, it could have saved the driver's life.<p>Usually I don't complain much about the gov, but just look at the construction mess they've created on 101, it's been like that for more than 4 years!
Tesla is playing with lives and should stop offering Autopilot.<p>Anything less than L4 autonomous driving is completely reckless. Calling it "Autopilot" when it's an L2 system should be criminal.<p>Yes, it's an extreme stance, but we're going to have a really hard time getting out true autonomous driving when companies are playing around with people's lives. You can say "well they know the risks" but it's not a closed situation. There are others on the road who will also die because of Autopilot mistakes.
I hear people say “self driving cars don’t have to be perfect, they just need to be safer than humans.” Here is a good example of why that might be harder to achieve than expected. Apparently the car had gotten confused at that exact location repeatedly before. That’s what self driving cars are going to do—if there is something “weird” it’s likely that every self driving car (at least, every one from the same manufacturer) that encounters the weird scenario will run into a problem. That could result in catastrophic failure modes at scale.<p>What would have happened if every car on that road had been an identical Tesla? How many crashes would have happened? How long would it have taken Tesla to issue a fix? How many miles of perfect driving would be required to make up for the cluster of crash events due to that one anomaly?
So, in a traditional embedded system, discovery can reveal the source code to help determine what went wrong (e.g. divide by zero error in x-ray machine). What are the lawyers going to get when they start looking into the Tesla's autopilot software?
This person died because of Tesla's cocky marketing, which leads people to believe that "Auto Pilot" is just that and that does very little to discourage this interpretation even though it is life-threatening. In this context them blaming it on the accident victim is a 100% asshole move.<p>I am life-long Musk's fan, but Tesla trying their hardest to weasel out of any responsibility here is incredibly damaging to their reputation. The future <i>will</i> come, trying to accelerate its arrival at all costs is reckless.
Tesla made some very big claims in their recent autonomy day event- basically, they claimed that they are years ahead of competitors, while operating on "hard" mode (no lidar). And yet, a number of participants in the short autonomy day demo rides claimed that the support driver had to disengage the autopilot.<p>Has Tesla provided any evidence which shows evidence that they are in fact far beyond competitors?
English is not my first language. I am confused by the blatant use of the term "Auto Pilot". Does it not suggest more automation than is currently feasible? Why not intelligent assist? Why is tesla/musk getting a pass here?
Do they not have Rumble strips in the US?<p><a href="https://en.wikipedia.org/wiki/Rumble_strip" rel="nofollow">https://en.wikipedia.org/wiki/Rumble_strip</a><p>I have been in situations in my youth where I was commuting or travelling tired, and nothing alerts you like the loud rumbling sound, and I'm sure the autopilot could detect it and bring the car to a stop if the driver hasn't been alerted by the in car warning system or the rumble paint...
I know the argument is always that autopilot is not intended by Tesla to be abused (never mind the CEO on national television abusing it).<p>However, there's what you tell humans, and then there's pragmatism regarding human nature. We should consider the latter.<p>It's possible to be both academically correct on the warnings/instructions given and also practically wrong on human psychology.
It seems like a (relatively) trivial addition to the Autopilot system would be to allow the drive to tell the car when it has made a mistake and if Autopilot(s) consistently make mistakes in the same area then Autopilot should give a specific warning, or force a disengagement, when it detects that it is approaching that area.<p>I can't imagine that this isn't already a thing or that someone at Tesla hasn't come up with this... am I missing something here?
The crux of the issue will be that autonomous capabilities will/have made cars overall safer. Tesla will be able to cite data and circumstances where the safety features saved lives, and likely more times than lives were lost.<p>The problem is that the lives that are lost when the safety features fail are different lives than would have been lost without it at all. The families of those killed in this manner will have their day in court.
can we just make a law that the driver is responsible/liable for the car they drive, regardless how they drive it? I mean breaks failing while you are actively driving is one thing, but to completely surrender control, that is a choice.
That news about this case is being spread everywhere irks me a bit, because people aren't considering the actual circumstances of the accident. Excerpt from the article:<p>"According to the family, Mr. Huang was well aware that Autopilot was not perfect and, specifically, he told them it was not reliable in that exact location, yet he nonetheless engaged Autopilot at that location. The crash happened on a clear day with several hundred feet of visibility ahead, which means that the only way for this accident to have occurred is if Mr. Huang was not paying attention to the road, despite the car providing multiple warnings to do so."<p>This deserves a WTF. He understood autopilot makes errors, complained to his wife several times [0] that the car usually makes errors in that exact spot, and yet wasn't paying enough attention on a clear day with ideal driving conditions to commute safely.<p>[0] "Family members say he complained about his Tesla veering into the same barrier at the exact location of the crash and that he brought his car into the dealership several times to report a problem with the autopilot function." from <a href="https://sanfrancisco.cbslocal.com/2019/05/01/family-driver-died-tesla-autopilot-crash-files-lawsuit/" rel="nofollow">https://sanfrancisco.cbslocal.com/2019/05/01/family-driver-d...</a>
Tesla should get taken to the cleaners for this, I <i>hate</i> that these cars w/"autopilot" are operated on the same streets I use.<p>They should focus on making quality, well-performing electric cars. Stop using us all as beta testers for an autonomous future most of us never asked for.
Living around silicon valley with all the tesla drivers here, you learn real fast it makes for some of the worst drivers out on the road. Regularly see them running red lights and just generally not paying attention. They crutch too hard on the auto-pilot and dick around on their smart phones, which is probably exactly what this guy was doing. It's bad for tesla drivers and it's bad for drivers around them. Hope Tesla loses this lawsuit just because of that.
I think self driving cars should have both high dynamic range cameras and LIDAR and maybe time of flight cameras. Input from a LIDAR system would be much more likely to detect that barrier, and computer vision via a camera much more likely to be fooled. I think an investigation into why the computer vision system failed to detect a barrier under clear daylight conditions will show the negligence on the part of Tesla. Lane lines are frequently not well marked, and sunlight glare is a difficult problem for cameras. However, you have to be able to detect a concrete barrier, in the worst of conditions. Does Tesla have in place some kind of determination of its lane detection accuracy and then alert the driver that it is turning off auto-pilot when accuracy is low?
I think it's unacceptable for automation to produce a worse result than a human in the same situation, with the same information. i.e. it's not acceptable for automation to fail danger, it must fail safe, including even if all it can do is give up (disconnects, warning tone, hands control over to the human driver).<p>I think it's reasonable, in the narrow case where primary control is asserted and competency is claimed to be as good or better than a human, to hold automation accountable the same as a human. And in this case, if this driver acted the way autopilot did based on the same available information, we would say the driver committed suicide or was somehow incompetent.<p>I see this as possibly a case of automation committing involuntary manslaughter (unintentional homicide from criminally negligent or reckless conduct).
So many people here have hardons for tesla hate.<p>The guy fucked up, bad. Tesla is not at fault here.<p>>"According to the family, Mr. Huang was well aware that Autopilot was not perfect and, specifically, he told them it was not reliable in that exact location, yet he nonetheless engaged Autopilot at that location. The crash happened on a clear day with several hundred feet of visibility ahead, which means that the only way for this accident to have occurred is if Mr. Huang was not paying attention to the road, despite the car providing multiple warnings to do so."