[bdamm]

Yeah this is interesting. My Tesla M3 does similar behavior, and so I often am ready to punch the accelerator. In the Tesla, this is how you solve that problem. The driver's push on the accelerator contraindicates the AI's decision to slow down, and so the car follows the driver's direction.

Where it gets dicey is the scenario where the "imminent collision" (hazards on, seatbelts tightened) detection is triggered, and the driver continues to push hard on the accelerator. Tesla has a fairly lengthy statement in the manual about this scenario. The bottom line is there are all kinds of heuristics at play that may or may not result in an override depending on the specific sequence of events.

[yread]

I'm amazed by people like you. You're a programmer, you know how your code looks like. Worse still, you have seen other people's code, how they fail to account for corner cases, you've seen so many articles on HN's front page about security bugs found by fuzzing. Yet, you trust your LIFE to "heuristics"? Do you really trust that when the proverbial black swan flies in front of your car the software won't swerve you into the oncoming traffic?

Why don't you just drive yourself or take a taxi?

[bdamm]

The "collision imminent" scenario would occur whether or not the car is in self-driving. If the car manages to avoid a collision that is the amazing part to me. And there's plenty of evidence that the Teslas do, in fact, avoid quite a lot of collisions. It would be foolish, however, to drive like it's going to resolve all your collisions for you.

I view these as assistance to driving. It's a comfort that the steering and brakes are not overridable. And honestly, if the system messes up so badly as to go flying into a barrier, well, that's not so different from a tire popping, another car careening across into yours, or other catastrophic and unlikely events that do happen. We have seatbelts, crumple zones, airbags, pre-tensioners, cargo hold-downs, and emergency services to help us survive what even 40 years ago would be unsurvivable accidents.

[perl4ever]

If the car avoids 99% of crashes, but crashes happen 1% of the time, and it causes crashes 1% of the time, then it's making you less safe.

Those are just arbitrary numbers and a simplistic framework, but the point is, you can have a huge increase in safety by the numbers, and a very small increase in problems due to the safety system that cancel it out, because the prior underlying rate of crashes was pretty small.

I think this is an abstract pattern that comes up in other contexts and it doesn't seem to be intuitive.

[bdamm]

True, the implementation quality does matter.

[yread]

Especially in a second world country like the US where semis dont have skirts and you do have crossroads without lights on highways

https://arstechnica.com/cars/2020/02/i-was-just-shaking-new-...