[slavik81]

I tried out a relative's Subaru on a 6h drive over the holidays. I really liked the adaptive cruise control for following behind folks who were not keeping a consistent speed. I just set it to a reasonable value at the maximum following distance and stopped worrying about my speedometer.

However, at one point the guy in front of me turned off onto a small side road. It was at night, and I don't think the car realized he had moved into a turn-off lane. It slammed on the brakes. I probably went from 90kph to 40kph before it realized I was not going to hit that car.

I completely failed to react to the situation. I was worried my erratic braking would cause an accident behind me, but in the moment, I didn't know how to stop it. That was not a type of emergency I had considered or prepared for.

[bdamm]

Yeah this is interesting. My Tesla M3 does similar behavior, and so I often am ready to punch the accelerator. In the Tesla, this is how you solve that problem. The driver's push on the accelerator contraindicates the AI's decision to slow down, and so the car follows the driver's direction.

Where it gets dicey is the scenario where the "imminent collision" (hazards on, seatbelts tightened) detection is triggered, and the driver continues to push hard on the accelerator. Tesla has a fairly lengthy statement in the manual about this scenario. The bottom line is there are all kinds of heuristics at play that may or may not result in an override depending on the specific sequence of events.

[yread]

I'm amazed by people like you. You're a programmer, you know how your code looks like. Worse still, you have seen other people's code, how they fail to account for corner cases, you've seen so many articles on HN's front page about security bugs found by fuzzing. Yet, you trust your LIFE to "heuristics"? Do you really trust that when the proverbial black swan flies in front of your car the software won't swerve you into the oncoming traffic?

Why don't you just drive yourself or take a taxi?

[bdamm]

The "collision imminent" scenario would occur whether or not the car is in self-driving. If the car manages to avoid a collision that is the amazing part to me. And there's plenty of evidence that the Teslas do, in fact, avoid quite a lot of collisions. It would be foolish, however, to drive like it's going to resolve all your collisions for you.

I view these as assistance to driving. It's a comfort that the steering and brakes are not overridable. And honestly, if the system messes up so badly as to go flying into a barrier, well, that's not so different from a tire popping, another car careening across into yours, or other catastrophic and unlikely events that do happen. We have seatbelts, crumple zones, airbags, pre-tensioners, cargo hold-downs, and emergency services to help us survive what even 40 years ago would be unsurvivable accidents.

[perl4ever]

If the car avoids 99% of crashes, but crashes happen 1% of the time, and it causes crashes 1% of the time, then it's making you less safe.

Those are just arbitrary numbers and a simplistic framework, but the point is, you can have a huge increase in safety by the numbers, and a very small increase in problems due to the safety system that cancel it out, because the prior underlying rate of crashes was pretty small.

I think this is an abstract pattern that comes up in other contexts and it doesn't seem to be intuitive.

[bdamm]

True, the implementation quality does matter.

[yread]

Especially in a second world country like the US where semis dont have skirts and you do have crossroads without lights on highways

https://arstechnica.com/cars/2020/02/i-was-just-shaking-new-...

[wffurr]

Same thing, first time I used the Subaru EyeSight system. Now I know to pay attention for that particular failure mode, and override with the gas pedal and a little steering.

Definitely surprised the heck out of me though the first time the car slowed way down on the interstate because the car ahead of me pulled onto the off-ramp.