[amelius]

What will happen if a deep-learning bug is found in Tesla autopilot where it misclassifies a fire truck for a bridge?

Will they ground all the Tesla cars remotely? Or disable autopilot remotely? Until they have gathered new training data and updated the software?

[Iolaum]

To add context. In Aviation, if they do find a bug in the software running an airplane's engine they do ground planes until they know more about impact and mitigation.

Not saying cars should do the same, just that it's not absurd to consider it.

[hef19898]

Well, to be honest something like Tesla's Autopilot would never have been certified in Aerospace anyway. And yes, the question of what authorities would do in such a case is a valid one. It seems like authorities are struggling with functions like that, would it be simply hardware, e.g. faulty airbag sensors, recalls would have been already be issued.

[ClumsyPilot]

" It seems like authorities are struggling"

I would rather describe it as sleeping at the wheel, passed out drunk and having pissed their pants after vomiting a bottle of hard liquir.

Okay, maybe i got carried away with the metaphor, but you get the idea

[hef19898]

Funny how someone on the internet can translate my understanding of "struggling" into plain English! Appreciate it!

Disclaimer: I really try to work on my ability to word stuff more diplomatically. HN is a good training ground for that!

[leesec]

It's funny how super pro regulation people never consider the lives lost caused by slow regulation. If Tesla solves driving it will lead to tens of thousands of lives saved a year. Since 2015 there have been 20 deaths in 16 accidents related to autopilot, globally. Doesn't sound like a lot in comparison to highway death numbers, over 6 years.

[jcranmer]

On a per-mile-driven basis, Tesla's self-driving accident rate is somewhere in the region of 5-10× that of human drivers, IIRC. And keep in mind that the self-driving here is already in a state of selection bias where it's more likely to be used in safer conditions (i.e., fully-grade-separated highway driving in clear conditions, rather than dense urban environments in inclement weather).

[Domenic_S]

Maybe you were thinking the autopilot accident rate being 5-10x _less_ than human drivers?

> In the 2nd quarter, we recorded one crash for every 4.41 million miles driven in which drivers were using Autopilot technology (Autosteer and active safety features). For drivers who were not using Autopilot technology (no Autosteer and active safety features), we recorded one crash for every 1.2 million miles driven. By comparison, NHTSA’s most recent data shows that in the United States there is an automobile crash every 484,000 miles.

Tesla publishes these safety reports; the accident rate has held fairly steady quarter over quarter: https://www.tesla.com/VehicleSafetyReport

[hef19898]

It's funny how people ignoring regulation, that developed during decades for very valid reasons, usually do so to further businesses and projects that benefit greatly by doing so. Tesla is one example when it comes to self-driving, Uber another one when it come to the Taxi business. All for the greater good of course.

[leesec]

You think Uber doesn't provide a good service to the world? You are in favor of Taxi/Medallion monopolies?

[6gvONxR4sf7o]

> If Tesla solves driving it will lead to tens of thousands of lives saved a year.

That’s a big if. If we solved aging, we’d save nearly all lives, so please don’t regulate my medical practice. I expect to have aging solved and millions of treatments rolled out by 2020.

[mullingitover]

Solving aging would be a massive win for humanity at the individual level, and a horrific catastrophe at the species level.

[BurningFrog]

If new technology X saves 100 lives for each person it kills, it will be banned.

Because the dead person has a name, a tragic story, and grieving relatives, which will make the news.

The 100 saved people are anonymous, don't know they were saved by X, and this will be unreported outside of obscure research journals.

[ClumsyPilot]

If you save a child from drowning you don't get a 'a kill coupon' you can retrieve next time you are accused of murder.

[ClumsyPilot]

Are in favour of medical experiments on random strangers?

The same argument applies - sorry you died, but the data might save someone else's life down the line.

A lot of unique medical data was produced by Nazis doing cruel experiments on people.

https://www.bu.edu/articles/2019/learn-from-nazi-medical-res...

[Consultant32452]

I’ll be happy if “AI” cars are safer than human drivers on average. Zero accidents/deaths can come later.

If everyone was flying their personal planes around and constantly banging into each other a minor software bug would not ground any planes. It would be more like a malfunctioning airbag recall.

[zibzab]

I can already see Elons angry tweets if MVA ever even thinks about that. Remember the Corona tweets?

What we need is a big fat lawsuit, since the government will not do anything to anger the only profitable US automaker.

[taneq]

That’s the theory but the recent issues with Boeing show that practice doesn’t always follow.

[hef19898]

Boeing lobbied the FAA for way too long to avoid grounding of the fleet. It did happen so, after it turned out Boeing's SW fixes didn't work.

Also, just how that part of the 737MAX was certified is the real scandal, if you ask me. This is the literal example proofing the rule, so.

[gameswithgo]

This has already been happening but self driving enthusiasts just keep thinking it is great. And regulators have done very little and otherwise smart people like Sandy Munro get extremely mad at what little regulators have hinted at. It is wild times.

[dreamcompiler]

Munro sat down with Musk and enthusiastically praised FSD but complained about how roads were often improperly painted. Musk--to his credit--said the car needed to work safely even in the presence of paint errors. (Video is on Munro's YT channel.) Munro is very critical of many aspects of car design; I cannot understand why he's such a fan of FSD.

I tried FSD on the $200/month plan and dropped it: It makes the car unsafe. To command a lane change you hold down the turn signal stalk. If you fail to hold it down long enough the car suddenly swerves back to the lane it was in. This is (to say the least) disconcerting at 80 mph.

FSD can also suddenly decide to do weird things that are difficult to correct even when you're paying close attention. It's unnerving. Ordinary autosteer (which is included with every Tesla at no extra charge) works well enough for me and it fails in more predictable ways; it's easy for me to build a mental model of its limitations. I'll stick with that.

[lastofthemojito]

> Munro is very critical of many aspects of car design; I cannot understand why he's such a fan of FSD.

Maybe due to personal profit?

https://old.reddit.com/r/RealTesla/comments/kxj0or/twitter_s...

[gamblor956]

TLDR: Munro admits to owning a bunch of Tesla stock, and there is a drastic change in his opinions of Tesla before he owned Tesla stock vs after.

[mikestew]

there is a drastic change in his opinions of Tesla before he owned Tesla stock vs after.

I'm confident that this would apply to just about anyone that has ever bought, and then sold, Tesla stock. Because, and I don't mean to overstate the obvious, if one's opinion didn't change then why sell the stock?

[nomel]

Or, perhaps there is a drastic change in his stock ownership after his opinions changed.

[leesec]

>I tried FSD on the $200/month plan and dropped it: It makes the car unsafe. To command a lane change you hold down the turn signal stalk. If you fail to hold it down long enough the car suddenly swerves back to the lane it was in. This is (to say the least) disconcerting at 80 mph.

This isn't true. You just tap the stalk down until it clicks. Sometimes if it detects issues in the other lane it will not complete the lane change and go back into your existing lane. But you do not have to hold it down the whole time.

[justapassenger]

> Munro is very critical of many aspects of car design; I cannot understand why he's such a fan of FSD.

Because he doesn’t understand it and thinks it’s magic. He has not much clue about modern tech and is just buying into the hype.

[moduspol]

> I tried FSD on the $200/month plan and dropped it: It makes the car unsafe. To command a lane change you hold down the turn signal stalk. If you fail to hold it down long enough the car suddenly swerves back to the lane it was in. This is (to say the least) disconcerting at 80 mph.

That's not quite accurate. Unrelated to autopilot / FSD, you can do a small press on the turn signal and it will signal three times and then stop signaling. You can also push all the way down and it will signal until you turn it off. You don't have to hold it down.

FSD will only continue switching lanes while the turn signal is on, so if you do a small press down, you may see the behavior you described.

[dreamcompiler]

To clarify I'm talking about a lane change initiated by me rather than merely allowing one recommended by the car. The latter only requires a brief flick of the turn signal stalk.

But the former required me to actively hold it down for several seconds--much longer than a full turn signal would require. I had to hold it down until the car was completely within the stripes of the adjacent lane or the car would immediately swerve back into the original lane. I tested it many times on a traffic-free road. Might have been a setting; I don't know.

[johnsimer]

Interesting. Either way I only have to tap the stalk down.

[texasbigdata]

Had the opposite experience on a testdrive purposely (within reason) trying to see if it would merge into a lane that subsequently had a merging onramp with a car resulting in the blindspot likely needing to move in.

Was super impressed it seemed to over correct and be thoughtful about consequences a few seconds down the road.

Anecdote though.

[revscat]

It is great. I own a Model S with autopilot, and have since early 2018. I have driven it across some dozen states, and two Canadian provinces. It is far perfect, but it is very, very good, and makes long drives much less stressful and tiresome.

And there have been at least two incidences that I can recall where Autopilot saved me from a wreck.

I would not choose to go back, and would buy it again without hesitation.

[virtue3]

We didn't get the FAA regulations we have now until after a ton of people died in aviation accidents and the aviation industry figured they need to get their image together.

So... yeah it's gonna take a lot of deaths for this to get regulated. It's a shame cuz we already basically know how to regulate this.

[drcode]

The same thing that happens to a human that is at risk of misclassifying a fire truck for a bridge: Once the odds of this happening are low enough (say 0.00001%) they can pass a driver's test and we give them a drivers license.

[tguedes]

The problem is that's 1 person. For Tesla's autopilot, that's the hundreds of thousands of cars running that broken model. The scale of the problem is much larger.

[play2computers]

I think that should very much depend on the severity and the statistical impact of the bug. As long as the long doesn't raise the deaths-per-mile-driven number above what humans would be doing, I would say it would be ok to just leave it running and just update once it is fixed. The goal is to get cars driving better than humans (which btw they already do by a huge margin, even with the fsd beta). so disabling the autopilot because of a potential bug would be condemning a lot more people to death than leaving it in. Interesting variation of the trolley problem i guess

[username_my1]

The idea is to have self driving safer than our average driver who’s always texting and driving, sometimes driving under the influence and sometimes just too stressed to pay attention to the road.

Self driving cars will make fatal mistakes but I have no doubt that tesla very soon will be able to be safer than the average driver.

Plus the more autonomous vehicles on the road the more safer it is.

Finally currently traffic accidents are the leading cause of death for 30yo so we aren’t exactly replacing a perfect system

[mplewis]

Ok. But this idealism doesn’t work when Tesla has clear failure modes that simply shouldn’t happen during normal driving.

Humans generally do not give their cars haircuts by slamming them under a stopped cargo truck. They do not generally smash straight into stationary emergency vehicles.

We can’t handwave basic safety issues away by saying “in aggregate, they perform better than humans in most conditions.” The basic safety issues get people killed. Leaning on some “average driver” fallacy is a way to ignore issues core to the tech stack.

[dreamcompiler]

Slight quibble: I very much agree with your overall point but human-driven cars slam straight into rescue vehicles all the time. They're often intoxicated but not always. There's just something about flashing red lights that attracts driver attention and all too often the car follows.

This is why we put the BRT (big red truck) behind the rescue scene and park it at a 45 degree angle. It weighs 15x as much as a car so it won't move much when a car hits it, and we want the car to bounce off sideways and away from the rescuers (us) when it happens.

[spywaregorilla]

Assuming that in aggregate they do kill fewer humans, then this is a rare instance of an actual trolley problem. Do you want to kill fewer people by changing the norms and reasons?

[kevingadd]

Big assumption to make if it turns out the numbers and data don't scale up once you do large-scale deployment!

[spywaregorilla]

You're not wrong. This should be done with great care.

[username_my1]

why doesn't it work? isn't all our society always abstracting away the individual stories behind statistics? (crime is down by 10% ! GDP is up by 0.2%, birthrate is down by xx).

we're used to it and we're happy about it and we use it to make decisions (buy house now, move to a new city ...).

Plus in those rare cases that you mentioned at least the same system will be feedback the incident to make sure that this won't happen again in the future... so in a morbid way there is a way to learn from such fatal incidents ... you can't really say that about such incident if the driver was a human.

[enchiridion]

It’s not a fallacy, it’s a risk analysis that will save lives.

[revscat]

This seems to be a reasonable take to me. Systems like Autopilot make driving safer. I'm worried that the conventional wisdom on HN is (a) that systems like Autopilotmust be perfect, which will never be attained, and (b) extremely loathe to recognize the times Autopilot has saved lives.

[londons_explore]

Even when a critical problem is found, I can't think of any example of cars being disabled remotely - either through technical means or laws saying 'you may not drive this car'.

[amelius]

Another option is that they recall the cars.

It's not uncommon: https://en.wikipedia.org/wiki/2009%E2%80%932011_Toyota_vehic...

[3dfan]

By the time people are allowed to let their cars drive unsupervised, the crash rate of AI versus human will probably be 1:10 or so.

So even when those cars ram into fire trucks from time to time, it would be better to let them do their thing. Otherwise people will grab the steering wheel, drive drunk, sleepy, angry etc and ram into all kinds of things again.

Currently, there are 6 million car accidents per year in the USA. Almost 100 people die in car accidents every day. So there is a ton of data to make the decision.

[moralestapia]

>By the time people are allowed to let their cars drive unsupervised, the crash rate of AI versus human will probably be 1:10 or so.

This sort of statement keeps being parroted over and over again. As Linus would say, talk is cheap, show me the code; then we can speak.

>So even when those cars ram into fire trucks from time to time [...]

This is just insane, honestly, and if this is the premise that guides the development of these sort of systems I'll be glad to never set foot on one.

[shkkmo]

The entire way we approach driver liability in general is insane.

The Attorney General of South Dakota was looking at his phone, swerved into the shoulder, killed man and then left the scene. He claimed he thought he hit a deer, even though the victim's head went through the windshield and the victims glasses were later found inside the car.

What consequences did the Attorney General face? Was his licence revoked or suspended? Did he serve any jail time? Did he resign? The answer to all of these is "No." The only result was two misdemeanors and a 500 dollar fine.

So yes, accepting occaisonal inhuman errors from a system that is 10x safer (hypothetical, no current systems have this record) than human drivers may also be insane but it would still be far more sane than the current approach to human drivers.

[moralestapia]

The problem with this is you're using the worst possible examples of driving as a benchmark, and that is quite a low standard to have.

Take a look at the best possible examples, instead, and try to improve over that, then we're talking.

You cannot be a shitty doctor and bring out an excuse like "Oh, but wait, this guy has lost more patients than me, so I'm fine".

You really don't want to have a race to the bottom where human lives are at stake.

[shkkmo]

I'm not using it as a benchmark at all.

I'm using a high profile example to point out clear issues with how we handle human driver responsibility. If your negligence directly results in you killing someone, you should lose your license to drive.

If we someday reach the point where autonomous driving systems are actually 10x safer than human drivers and those systems still have issues with hitting emergency vehicles, we should absolutely hold those companies responsible for those accidents.

My point is that not holding those companies responsible would be less insane than our current practice of letting clearly negligent drivers continue to drive with minimal consequences after they kill someone.

So instead of being outraged about this hypothetical future, why not be outraged about the insane lack of consequences that drivers face currently?

[kevinventullo]

That seems more like plain old political corruption than a problem with the way driver liability is handled. If you or I did that we’d almost certainly be facing prison time.

[pfortuny]

There is no code, that is the problem.

You (or the Transport Administration) do not have access to the training data, the training parameters or anything at all.

It is just a black-box which we are all to trust because "it works, mostly".

In my book, that goes against any notion of admissibility by a government agency.

[amelius]

Yes anything or anyone outside of the norm becomes a target.

E.g. wear your hair in a funny way that's never been trained on? You're a target!

[croon]

In aggregate, humans have a lot of failure modes when driving, but it's also difficult to compare aggregate data with specific AI failure modes.

I have been driving for almost two decades with 0 accidents. I'm not saying I can't have a lapse of judgement or do something stupid going forward, but I certainly won't mis-class an object, nor kill myself over it.

I hypothetically want bad drivers to be replaced by AI because it's likely already better. But replacing everyone with AI (at the current generation of AI, which isn't the first, nor the last) will undoubtedly lead to tons of avoidable deaths, and I'm not keen on drawing a lottery ticket for it.

[jacobr1]

I'm more interested in replacing _other_ drivers, more than myself. Really if we could replace the bottom 10% of drivers with AI, even at the level we have today, I imagine that would be a net improvement. But that isn't really a feasible program. As for future, improved AI, I would trade my own driving for the more efficient and safer system.

[Domenic_S]

That's really the move, having another class of driver license that lets you manually drive. Anyone can get an in ai self-driving car, but if you want to manually drive you have to pass a difficult skills test to prove you could outperform the ai.

[kevingadd]

People already let their Teslas drive unsupervised, they're just "not supposed to". That will be increasingly permitted over time, either implicitly or explicitly. It's not a switch that will be flipped nationwide once the data hits a threshold.

[leesec]

Misclassification issues don't really matter in driving. Sometimes I mis-classify the things I see, all that matters is if I keep driving safely.