[im3w1l]

This is my biggest fear with self-driving cars. Correlated failures. As a society we are extremely good at dealing with independent accidents. We can calculate very precisely how many people will die of traffic in a given year and we can account for it, we can have insurances, and we can decide exactly how much we are willing to spend to save a life on the margin.

But if everything is fine, everything is fine, everything is fine, and then all hell breaks lose? We are not as good at dealing with that.

[brk]

My fear is similar, but more along the lines of adversarial attacks as various weaknesses are exposed. Imagine people taking advantage of zero-day exploits that cause self driving cars to veer off the road, stop suddenly, collide, etc. It is really not that far fetched. This technology is very far away from maturity, IMO.

[Ukv]

Relevant XKCD: https://xkcd.com/1958/

[nytesky]

Your run of the mill hacker sure, but adversarial nations about to invade another nation and want to keep the Western World occupied? DC was brought to a crisis by the sniper; can you imagine if cars suddenly started plowing through school pickup lines or concert venues with no way for occupant to stop it? Or even without an occupant?

[nicbou]

On the other hand, the cars were recalled, a team will study what happened, the problem will get fixed, simulations and tests will be created.

When I make a dumb mistake, the other drivers rarely learn from it.

[isthatafact]

As someone else hinted, there may be nothing to "fix", but rather this seems like a specific situation that it had just never encountered before. Adjusting the model to cause a safe response to that particular single rare situation (either manually or by accidental training) does not solve the apparent problem that the machine is not able to comprehend the world.

At least that is how I (as a non-expert) imagine these models work -- the model has an excellent chance of crashing at every new unique situation it encounters out of a nearly unlimited set of possible situations (which implies a high frequency of encountering new situations).

[Sohcahtoa82]

Even if it's not something they specifically fix, I wonder if the scenario could get added to some test suite?

[simion314]

So in future your self driving car might be recalled at any random time because a new corner case from an infinity was found. If the car can't be driven by a human all owners of thsi cars will be stuck wherever they were at that moment.

[roxgib]

This will be interesting to watch. If I bought an autonomous car and the autonomous mode was disabled for a few days or even weeks while a bug was fixed I can always fall back to driving it myself. If that's not possible (maybe because the car doesn't support it or I didn't have a licence or human drivers had been banned) and suddenly it's a whole different situation. Of course owning a car might become less common in itself, if you're just taking them like an Uber you can always switch to a different company.

[leoh]

This will have to be priced in to insurance.

[add-sub-mul-div]

Yeah, there's nothing akin to a software update that would cause the entire fleet of human drivers to start driving badly or unexpectedly all at once.

We also know how to hold individuals accountable for independent accidents. We know we won't get justice when people will inevitably start to get killed by standard corporate greed, incompetence, enshittification.

[1000100_1000101]

> there's nothing akin to a software update that would cause the entire fleet of human drivers to start driving badly or unexpectedly all at once.

All, no. Enough to make a difference, easy.

  Black Friday in February!  In-Store offer only!
  $99 Playstation 5 to first 100 customers at each Walmart location!

You can bet there will be a significant increase in people driving badly. edit: Make it Taylor Swift tickets, and you can increase the size of the frenzy.

[magnetowasright]

Not even a cheap console or taytay tickets needed. I once had a lady with 5 kids in the car go to ram me when I went around her to get into the car park while she was waiting in line for the KFC drive through. Nuts lol

[gdestus]

I imagine this can be remedied by slow rolling non critical updates out so that the entire fleet doesn't get upended by Bobby Tables at once. You could trivially observe the daily change in accidents/collisions/whatever and adjust fire

[shermantanktop]

Works great until the problem condition is not evenly distributed in place and time. Imagine that the release goes out in June but can’t handle icy roads; or the release goes out and can’t handle leap years; or it goes to cars in Iowa but has a problem interpreting ocean mist.

[kevinventullo]

That remedy already applies to massive profitable services like Google and Facebook, yet they still have outages caused by sloppy configuration pushes.

[hyperman1]

I notice, when e.g. it rains enough for long enough, a lot if people get annoyed and start driving badly. You get some kind of contagion where everybody gets annoyed and tired because all other drivers are annoyed, tired assholes. Sometimes it even persists after the bad weather, especially if there was gridlock and/or end of workday. Humans do have a (lighter) version of collective bad driving.

[weaksauce]

that's one reason crash traffic happens and stays for hours after the original reason for the traffic to happen in the first place is gone. I remember reading that one person applying their brakes too hard can have a knock on effect causing a traffic pattern to emerge for a long time after the initial braking ever occurred.

[bigyikes]

Yeah, I wish this was emphasized more in driving school. I wonder how much traffic could be reduced just by making people conscious of their traffic-causing behaviors. (for example, using your brakes to slow down when there is nothing in front of you and others behind you)