[emn13]

Actually, all kinds of data pertaining to the decision making process is recorded (at least for some of Tesla's competitors, not sure about Tesla), and in great detail. The data is specifically designed to make the AI driver "debuggable", i.e. it includes all kinds of details, intermediate representations, etc that an engineer would need to improve a poor decision, and thus certainly to understand a poor decision.

Whether that kind of logging is always on or was specifically on here, I don't know, but I'd expect Tesla can analyze why this happened: the car does have the ability to explain itself; it's just that owners and drivers do not have access to that explanation.

[mrguyorama]

You can log and debug the inputs going into the black box, and the outputs, but how do you debug inside the black box?

[emn13]

The box isn't as black as you might think; they're not training some monolithic AI model, there are separated systems involved. Also, the models aren't entirely freeform; i.e. engineers embed knowledge of how the world is structured into those networks.

They can use those intermediates to project a kind of thought process others can look at - and you've probably seen videos and images of that kind of thing too; i.e. a rendered version of the 3d intermediate world it's perceived, enhanced with labels, enhanced with motion vectors, cut into objects, classified by type of surface, perhaps even including projections of likely future intent of the various actors, etc.

Sure, you can't fully understand how each individual perceptron contributes to the whole, but you can understand why the car suddenly veered right, what it's planned route was, what it thought other traffic participants were about to do, which obstacles it saw, whether it noticed the pedestrians, which traffic rules it was aware of, whether it noticed the traffic lights (and which ones) and how much time it thought remained etc.

...at least, sometimes; I don't know anybody working at Tesla specifically.

Here, for example waymo has a public PR piece kind of highlighting all the kind of stuff they can extract from the black box: https://blog.waymo.com/2021/08/MostExperiencedUrbanDriver.ht...

And while they emphasize their lidar tech, I bet Tesla's team, while using different sensors, also has somewhat similarly complex - and inspectable - intermediate representations.

[mcguire]

"Also, the models aren't entirely freeform; i.e. engineers embed knowledge of how the world is structured into those networks."

In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6.

"What are you doing?", asked Minsky.

"I am training a randomly wired neural net to play Tic-tac-toe", Sussman replied.

"Why is the net wired randomly?", asked Minsky.

"I do not want it to have any preconceptions of how to play", Sussman said.

Minsky then shut his eyes.

"Why do you close your eyes?" Sussman asked his teacher.

"So that the room will be empty."

At that moment, Sussman was enlightened.

(https://news.ycombinator.com/item?id=10970937)

IIRC, in the incident where the Tesla [Edit: Uber self driving car] collided with a pedestrian pushing a bicycle in Arizona, the Tesla repeatedly switched between calling the input a pedestrian and a bicycle. And took no evasive actions while it was trying to decide.

[mrguyorama]

>the incident where the Tesla collided with a pedestrian pushing a bicycle in Arizona

That was Uber's self driving car program. Notably, the SUV they were using has had pedestrian detecting auto-stopping for several years, though I'm sure it's not 100%

[mcguire]

Sorry, Uber, you're right! Whoops!

[mcguire]

Does Tesla use a neural network for sensing and scene recognition/the observe-orient steps? For parts of the decide-act steps?

That particular kind of black box is very black; it has hundreds of thousands to millions of inputs feeding a hyperdimensional statistical model.