[ReidZB]

> Being able to effectively reason about what the automation is doing is such an important part of why these technologies have been so successful in flight, and examples like this illustrate how far off we are to something like that in cars.

Is that actually the case, though?

I would hope, although perhaps I'm mistaken, that the developers of the actual self-driving systems would be able to effectively reason about what's happening. For example, would a senior dev on Tesla's FSD team look at the video from the article and have an immediate intuitive guess for why the car did what it did? Or better yet, know of an existing issue that triggered the wacky behavior?

Even if not, I'd hope that vehicle logs and metrics would be enough to shed light on the issue.

I don't think I've ever seen a true expert, with access to the full suite of analytic tools and log data, publish a full post-mortem of an issue like this. I'm certain these happen internally at companies, but given how competitive and hyper-secretive the industry is, the public at large never sees them.

[ChicagoBoy11]

They certainly are trying very hard, as far as I can tell. Tesla's efforts on data collection and simulation of their algorithm are incredibly impressive. But part of why it is so necessary is that there is an opaqueness to the ML decision-making that I don't think anyone has quite effectively cracked. I do wonder, for instance, if the decision to go solely with the cameras and no LIDAR will prove to ultimately be a failure. The camera-only solution requires the ML model to accurately account for all obstacles, for example. As crude, and certainly non-human as it is, a LIDAR with super crude rules for "dont hit an actual object" would have even at this point prevented some of their more widely publicized fatal accidents which relied on the algorithm alone.