Cruise Knew Its Self-Driving Cars Had Problems Recognizing Children

The title makes it sound like the car wasn't detecting children as opposed to misclassifying children as adult humans.

&gt; According to one safety memo, Cruise began operating fewer driverless cars during daytime hours to avoid encountering children, a move it deemed effective at mitigating the overall risk without fixing the underlying technical problem. In August, Cruise announced the cuts to daytime ride operations in San Francisco but made no mention of its attempt to lower risk to local children.<p>Edit: looks like Cruise is not long for this world. Insiders seem to have lost faith and are starting to leak. I&#x27;d bet it simply cannot withstand any serious scrutiny into its safety levels, practices, and culture.

There were clear signs that Cruise was rushing things with their expansion, but I didn’t know the performance was this bad. Based on developments from the past few days [1], their safety culture seems to be closer to Tesla’s than it is to Waymo’s.<p>[1] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=38002752">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=38002752</a>

If a (usually lead) construction engineer notices (or is told about) a fault&#x2F;design issue&#x2F;problem, knows that it is dangerous, and does nothing about it, then something bad happens because of that, in many cases and many countries, the engineer ends up personally responsible for the outcome.<p>This should be expended to other industries as well.

After hearing about how often their cars need a human to intervene, I started wondering about that incident where a Cruise taxi ran over a person that had been thrown into its path, stopped, and then started up again to move to the curb.[1]<p>Did the car stop, notify the mothership, and then have a person direct it to pull out of the flow of traffic? How would we know? If the car moved on its own, that&#x27;s bad. If the car moved after being contacting a &quot;remote assistant agent&quot;, that&#x27;s bad, too.<p>1 &quot;The Cruise car ran over her, briefly stopped and then dragged her some 20 feet before pulling to the curb, causing severe injuries.&quot; <a href="https:&#x2F;&#x2F;www.nytimes.com&#x2F;2023&#x2F;11&#x2F;03&#x2F;technology&#x2F;cruise-general-motors-self-driving-cars.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.nytimes.com&#x2F;2023&#x2F;11&#x2F;03&#x2F;technology&#x2F;cruise-general...</a>

It sounds more like a path prediction problem than one of not seeing them at all. I&#x27;m guessing they expected it to still handle them as well as a human driver would, even if it relied on the adult pedestrian logic.<p>I have doubts as to this being the right choice, but it seems less unreasonable than what I thought from the title.

&gt; “Our driverless operations have always performed higher than a human benchmark, and we constantly evaluate and mitigate new risks to continuously improve,” said Erik Moser, Cruise’s director of communications. “We have the lowest risk tolerance for contact with children and treat them with the highest safety priority. No vehicle — human operated or autonomous — will have zero risk of collision.”<p>Notice that this contains nothing of substance: no &quot;human benchmark&quot; is actually defined; nobody was asking if they constantly evaluate risks (do they also remember to breathe in and out every few seconds?); &quot;the lowest risk tolerance&quot; is meaningless, as is &quot;the highest safety priority&quot;; and obviously no vehicle will have &quot;zero risk of collision&quot;, just as no vehicle has zero risk of being hit by a meteorite.

I was kind of shocked in 2019 to see those cars driving around SF. Most jurisdictions would never allow this in a million years.<p>Why would you allow some tech company to develop its technology at the risk of pedestrian safety? What benefit does this have to the people living in these neighborhoods?

It&#x27;s also curious how quickly this gets voted off the front page because it doesn&#x27;t fit the &quot;AI eating the world&quot; narrative sufficiently.

This is the wider problem of AI in general. It is right some percent of the time. You can only use AI in situations where being wrong randomly, without anyone to sue or blame, is OK.<p>Why people keep trying to use AI to drive cars, I have no idea. It’s like starting with the hardest of hard problems immediately. Fools errand, if you work in self driving cars get out ASAP while the rest of AI is still hot and you can get paid. Because the other stuff is on cusp of the trough of despair given the increasingly desperate emails from salesmen I get from XYZ.ai startups.