[benzor]

It's really quite sad that the regulation of autonomous vehicles been so slow to come along. Public roads are filled with other drivers, passengers, and pedestrians that did not consent to be a part of a large scale beta test for partial driving automation that could fail at any time. I believe this is a case where self driving software should be default illegal until proven safe. Most companies in this industry, thankfully, seem to be moving carefully and rolling out their products conservatively; Tesla seems to think "move fast and break things" is an appropriate motto for 5000 lbs projectiles on public roads.

[wpietri]

I partly agree, but it's also sad that we need to add to the bureaucracy just to insist that professionals act professionally. As you point out, most companies are doing fine. But it wasn't just Tesla playing fast and loose; as far as I'm concerned Uber execs should be doing time for negligent homicide: https://www.npr.org/2019/11/07/777438412/feds-say-self-drivi...

[HWR_14]

> it's also sad that we need to add to the bureaucracy just to insist that professionals act professionally

That's how bureaucracy and laws form. Individuals (people or companies) do X. The rest of society doesn't like it. They outlaw or regulate it. That's why the rivers no longer catch on fire in the US.

[wpietri]

Sure, but I'll note the other companies are self-regulating on this. Regulation tends to slow progress, so what I'd rather see is what places like Waymo are doing: acting like responsible adults so regulation isn't forced upon the whole industry prematurely.

[studentrob]

Tesla is the big fish here. If Uber has one instance, then given the size of their program, Tesla may have hundreds.

[AKSucks]

Thing is, it wasn't just one incident, it was just one incident that resulted in a death.

When Ubers started self-driving it took just a few hours before there were videos on twitter and youtube of them driving right though red lights without a care in the world.

[studentrob]

Uber: 1 death and halted the program

Tesla: many deaths, no halt, no improvement in the program. See: phantom braking

[wpietri]

Let's not give them too much credit here. I think Uber's halt had more to do with Uber's change in CEO and the indictment of the guy who ran their self-driving car program. Plus the fact that it was a giant money sink with no short-term return being run in a company that has never been profitable and can no longer raise infinite investor money.

[studentrob]

I don't think it matters whether or not you think Uber was acting responsibly or reflexively. The point is their program is done while Tesla's, which has seen far more fatalities, continues unabated.

[bob33212]

If every driver on the road today was never drunk/distracted/enraged I would agree, but the reality is that humans driving cars kill other people every single day. We should fix this with a better system. Tesla and Waymo seem to be making progress. I don't expect them to be perfect but in the long run this will save lives.

[wpietri]

You seem to be implying that in the short run it's ok for them to kill some extra people. One, I don't think that's necessary; Waymo is a good example of how a safer approach is also apparently no less effective. Two, you're presuming that we will get to self-driving cars that are economically viable and safer than current human-driven ones, something that is not a given. And three, it's not clear to me who gets to decide exactly how many unwilling people should be sacrificed on the altar of technological progress, but I hope it's not us and it sure shouldn't be Musk.

[kjksf]

https://www.youtube.com/watch?v=zdKCQKBvH-A

Here's a video of Waymo car in Phoenix. Around min. 17 it gets confused by a cone and stops in the middle of two lane road, hugging both lanes.

Then google sends support but just when they arrive, the car drives away from them.

They were lucky it didn't lead to an accident and as long as they keep the fleet to 600 cars then yeah, the accident rate will be much lower than Tesla's AutoPilot, shipped in more than a million cars.

My point is not to rag on Waymo just to inject some reality.

We don't have a choice between "safe and unsafe" way of developing self-driving software.

We have a choice between "test software we know can't handle all situation on real roads and make it better based on that testing" or "we'll never have self-driving software".

Except the other option will rather be: "U.S. doesn't allow testing of self-driving software on real roads and a Chinese company will develop it and will capture a trillion-dollar market in U.S."

[elil17]

The Waymo car in that video drove extremely safely give that it was confused. It was conservative and thought it might have seen and obstacle so it stopped. Did this inconvenience other drivers? Absolutely. But it was not a major safety risk. In fact, slowly coming to a stop is the legal and correct thing for a confused or impaired human driver to do. In comparison, Teslas seem to rapidly and suddenly brake for no explainable reason while traveling at fast speeds and do so routinely. Further, Teslas have other safety issues which are indicative of sloppy design (such as the fly-by-wire passenger doors that will trap back sheet occupants in the car if the electronic system is disabled). This is a failure of Tesla specifically, and regulating to stop it wouldn’t really slow down others like Waymo.

[Reason077]

> ”Teslas seem to rapidly and suddenly brake for no explainable reason”

The reason is widely known. Phantom braking is caused by rogue radar reflections that confuse the car into thinking there’s an obstruction in its path, activating the AEB automatic braking.

The real question is, why does it happen more often with Teslas than with other cars equipped with radar AEB? Maybe Tesla’s is just more sensitive.

[hungryhobo]

heh, i guess their fix to that is to just get rid of radars altogether.

[wpietri]

Exactly. There's a big difference between approaching this problem with a "first do no harm" perspective and a "move fast and kill a few people" perspective.

And this part from the previous poster strikes me as a big problem: "They were lucky it didn't lead to an accident and as long as they keep the fleet to 600 cars then yeah, the accident rate will be much lower than Tesla's AutoPilot, shipped in more than a million cars."

That seems like an excellent reason to keep the number of active cars very, very small. Rather than, as stated, an excuse for shrugging at a death rate at least 1667 times higher.

[elil17]

100% agree! And I don’t think the approach needs to be “first, do no harm”. I would be very happy with “move at a normal pace and do your best not to kill anyone.”

But “move fast and kill people” is ludicrous and it’s exactly what Tesla is doing.

[heavyset_go]

> We have a choice between "test software we know can't handle all situation on real roads and make it better based on that testing" or "we'll never have self-driving software".

This is like saying that we'll never have a cure for cancer if we can't experiment on the public without their consent.

The bar for medication, at least, is proving safety first before testing on large amounts of people and allowing the public to buy it.

[gavinhoward]

> The bar for medication, at least, is proving safety first before testing on large amounts of people and allowing the public to buy it.

Except for vaccines, apparently...

[dataflow]

> We don't have a choice between "safe and unsafe" way of developing self-driving software.

It's also entirely possible Waymo hasn't achieved peak perfection in its software development practices despite doing better than Tesla, and that another entity could do it more safely.

[cbozeman]

How many unwilling people are sacrificed because we won't ban alcohol and all impairing drugs without exception and then enforce those bans with immediate, Judge Dredd-style summary execution?

I bet you that'd reduce traffic fatalities dramatically too.

How far do you want to go to 'save lives'.

I guarantee you with 100% certainty I can design a society that will 'save lives' at every turn for every single activity, and I can guarantee you with 100% certainty you wouldn't want to live in it.

[wpietri]

You seem to ignore that we've already tried banning alcohol and currently ban many drugs. We scrapped banning alcohol for a reason.

[fitzie]

you are arguing a fake hypothetical. Tesla accident data shows that their autopilot feature reduces accidents.

https://www.tesla.com/VehicleSafetyReport

fortunately our regulators don't take a one size fits all approach for how new technologies can be developed

[avereveard]

It doesn't, as it compare general rates with self selected "good driving conditions" as defined by the software - only highways, only good enough weather, only good enough maintenance state of the car.

[wpietri]

Your notion is that everything is fine because according to Tesla, a company with a leader known for telling whoppers, they are killing less people on net?

Even if we trust them on that stat, which I certainly don't, that still doesn't mean they aren't killing people unnecessarily.

[Swenrekcah]

If FSD is accurately described as “not always worse than a drunk driver” then please remove it from the roads.

[steelframe]

I can't withdraw my consent to share the road with intoxicated or distracted drivers. That's just a fact of life. That doesn't mean I shouldn't be able to withdraw my consent to share the road with Beta software.

Plenty of businesses are able to build very effective safety mechanisms for motor vehicles without subjecting the general public to Beta software. To me this is a case where the ends do not justify the means.

[davrosthedalek]

I understand your point, but I would frame it stronger: Indeed, we, as a community, have withdrawn our consent to share the road with intoxicated drivers. You break laws if you drive intoxicated. That beta software doesn't break any laws, but maybe it should.

[HWR_14]

> We should fix this with a better system.

What if we made driving tests far more difficult (otherwise, you can drive a 50(?)CC moped that tops out at 35mph)? What if we had government subsidize rides home after going out drinking? There are a lot of solutions that don't involve "let companies test 5000lbs autonomous vehicles on public roads." Heck, those companies have enough money let them buy a city and test it on the now private streets.

> Tesla and Waymo seem to be making progress. I don't expect them to be perfect but in the long run this will save lives.

Let's assume for a second your unstated assumptions I was alluding to above are correct. This is the only way to make a better system and that system has to be tested on public roads. Why do we allow two private companies to reap the inevitable huge monetary rewards when we all pay for that system with a higher increased risk of dying in the meantime?

[rantwasp]

nah. just because some drivers do stupid things does not mean that we should allow this "FSD" travesty. The thing is: if you're a human driver and you screw up you pay the consequences. It's codified in the law and you are fully aware that there are consequences if you don't follow the rules.

Now, if you drive one of these "FDS" cars and it ends up in an accident where people die, who is responsible? Are you responsible? Is the car manufacturer responsible? Do we just write it off as a freak accident with 1 in a million chance of happening again?

[mgarfias]

FSD != auto pilot

[rantwasp]

yes of course. fsd is only 3 letters but IMHO they’re both lies.

[_ea1k]

> Most companies in this industry, thankfully, seem to be moving carefully and rolling out their products conservatively;

Having used other products, I think this is objectively not true. I've seen "Pro" pilot accelerate itself into its own collision warning and randomly fail at basic curves.

I've seen video of Ford Copilot failing due to glare and jerking toward trees.

For some reason, Tesla just gets more attention. Some of the other beta-like behaviors are actually worse.

[AlexandrB]

Tesla gets more attention because their CEO keeps promising the world[1] and selling these features as "Full Self Driving" instead of "Copilot".

[1] https://mobile.twitter.com/elonmusk/status/68627925129377792...

[_ea1k]

The others claim that you can drive without your hands on the wheel, despite the fact that the time required for takeover can be milliseconds.

I'm really not convinced that is objectively better, even with whitelisted roads.

[amanaplanacanal]

I haven’t seen this, but I’m out of the loop. Which ones are claiming this?

[_ea1k]

GM SuperCruise right now, and Ford is claiming that their "Blue Cruise" product will be able to do it soon. There may be others, but those are the two that I'm familiar with.

I've also been observing a general trend of reviewers rating the performance of new vehicles based in part on how long it can go without nagging the user to hold the wheel. sigh

[beardyw]

Quite a lot of milliseconds in fact. For a visual stimulus it is 1/4 of a second minimum. That's 22 feet at 50 mph.

[TearsInTheRain]

I think Tesla should definitely be accountable for their software and damage that it causes but involving overly risk adverse regulations will push out self driving cars by a decade, maybe indefinitely.

[iaw]

Innovation always precedes regulation/safety. Look at the early automobile and the road marker[0]

[0] https://en.wikipedia.org/wiki/Road_surface_marking

[benzor]

I agree with your point that government regulation is fundamentally reactive to private sector innovation, hence lagging. That being said, this particular issue of autonomous driving has been a hot topic for the better part of a decade now and I would like our governments to tackle it.

[slg]

>I believe this is a case where self driving software should be default illegal until proven safe.

If we go with your suggestion, what would you consider "proven safe"?

Autopilot has been running in hundreds of thousands of cars, has driven several billion miles, and we still don't have enough data to prove whether it is safer or more dangerous then an average human driver. Accidents are rare and we therefore need a huge amount of data before we can be confident that the accident rates we are seeing are predictive. I have no idea how you collect that type of data without them being tested on real roads with other real drivers.

[dragontamer]

How about actually passing some basic tests before they're deployed?

(Warning: unnecessarily loud video for some reason): https://twitter.com/finance_degen/status/1307529357951467531

Lets start with passing these basic tests before selling them as working features. I don't think we're asking for a very high bar.

[slg]

There is zero context on that video that tells us what is happening there. We have no idea if that is the Autopilot failing or the emergency braking failing. We also have no control group to tell us what percentage of humans would stop short of that dummy. It is inexact due to Twitter's video player not showing fractions of a second, but it looks like there were approximately 2 seconds between when the dummy started moving forward and when it was hit by the car. The average human time to braking is 2.2-2.3 seconds[1]. Is the car even failing that test in comparison to a human?

It also isn't clear from watching that video what the safest and therefore desired behavior should be in that situation. A self driving car is obviously not going to prevent all accidents, so it is a question of minimizing potential harm. We don't want a car to aggressively brake whenever someone at a street corner takes a step towards the road. We therefore need to balance the chance of a person stepping into the path of the car with the risk of braking when it is unnecessary and causing a rear end collision. The problem in the linked thread is overaggressive braking so forcing the car to pass a test that rewards overaggressive braking would only make that specific example worse.

That leads back to my point about needing a huge amount of data. You can't just run a car through an obstacle course to know whether it is safer or more dangerous than a human. You need to have it interacting with unpredictable humans and you need to do it repeatedly before you can confidentially predict whether it is safer or more dangerous than a human.

[1] - https://copradar.com/redlight/factors/IEA2000_ABS51.pdf

[dragontamer]

> We also have no control group to tell us what percentage of humans would stop short of that dummy.

IIRC, Subaru and other companies pass these simple emergency braking tests 100% of the time.

That test was a Chinese test IIRC, but the software doesn't change between countries. Similar tests have been done here in the USA by insurance groups to set insurance rates, but a government-mandated test for what "emergency braking" really means (before you "sell the feature to the public", lets actually have a government-mandated test similar to that video).

You shouldn't be allowed to call your stuff "autopilot" or "full self driving", or "emergency braking" or "pedestrian avoidance" (or some other set of words) unless you can... you know, avoid pedestrians and emergency brake in a well-controlled test.

Avoiding balloon people is enough. But its a well known fact that Tesla repeatedly fails at these simple tests, when other groups (ie: Mobileye group / Mobileye hardware) manages to emergency brake in time.

----------

IIHS test: https://www.iihs.org/news/detail/performance-of-pedestrian-c...

The issue is that 3rd party non-government groups (ie: IIHS) are the ones running these tests. There's no advocacy group for US consumers as far as I can tell. IIHS is primarily about serving their master (insurance companies).

Don't get me wrong: IIHS is doing good work here. But its not their job to protect the consumer.

EDIT: I got my sources mixed up. Tesla apparently passed the IIHS test.

It was the AAA test they failed: https://insideevs.com/news/377427/video-tesla-model-3-failed...

[slg]

>IIRC, Subaru and other companies pass these simple emergency braking tests 100% of the time.

They do not pass 100% of the time unless you have a very narrow definition of "these simple emergency braking tests". No emergency braking system is foolproof.

>But its a well known fact that Tesla repeatedly fails at these simple tests,

You say this while at the same time the source you include has Tesla in the middle tier of results.

Either way, my point is not that Teslas are safe or that they perform well on this test. The point is that this test does not tell you whether a car being driven by Autopilot is safer than a human.

[dragontamer]

> They do not pass 100% of the time unless you have a very narrow definition of "these simple emergency braking tests"

Lets get them working consistently under well defined, standard, simple, emergency braking tests before worrying about the real world.

Like not hitting a balloon dressed up as a pedestrian during clear skies in sunny weather. I don't care about rainy days until we get the bright / sunny weather figured out.

[HWR_14]

> we still don't have enough data to prove whether it is safer or more dangerous then an average human driver.

I would point out that given the dangers involved with accidentally turning over a million vehicles into autonomous 5000 lbs missile, erroring on the side of caution seems fine. The benefits are quite low: if the autopilot had been on since inception between 20 and 100 lives would have been saved (I accept your "several billion miles" number and point out that that the average fatality rate is 1.1 per 100 million miles driven, but that is based on averaging in 40 year old cars with fewer safety features and shrinks ever year.). The costs could be astronomical: a simultaneous failure (security, mistraining, date bug, whatever) could result in hundreds of thousands or millions of deaths.

Which means there are three errors to consider. (1) Obviously, some things (bugs, exploits) are unknowns and there will always be an inherit risk there. I would say that these risks may forever make self-driving cars too risky. (2) It is difficult to come up with any actual test of driving skills. This is especially true because any test will suddenly become the target so we have to have the test cover everything. (3) Actual driving errors: Both of the above assume that the AI can drive as well as a person. That's obviously difficult to do. And we would need to see a huge improvement to justify adding a new risk factor.

[slg]

>The benefits are quite low: if the autopilot had been on since inception between 20 and 100 lives would have been saved

This is only the case if you look at the current system as the finished product. The biggest benefit is that it gets us closer to a true self driving system. That would not only save millions of lives, but it would revolutionize logistics and economics of transportation which can in turn reshape society.

>The costs could be astronomical: a simultaneous failure (security, mistraining, date bug, whatever) could result in hundreds of thousands or millions of deaths.

I have no idea what scenarios you are imaging that could lead to "hundreds of thousands or millions of deaths." Almost every Autopilot death in the US makes national news. There is no way hundreds of people could die without there being some type of intervention in the system.

[northwest65]

> Autopilot has been running in hundreds of thousands of cars, has driven several billion miles,

Has it really though?

> I have noticed for me at least it started happening after I updated at 2021.4.18.11

This implies that the functionality of the AutoPilot is constantly changing, presumably meaning each version has thousands of miles rather than AutoPilot having 'several billion miles'. It doesn't seem like you can trust past performance is the users are to be believed.

My assumption is that OTA updates won't be allowed once this stuff starts requiring certification.

[slg]

I think the first half of your comment is a pointless semantic debate. The Autopilot system has driven billions of miles. Those miles obviously all aren't equally relevant. The older miles lose value as the hardware or software changes. However those miles don't all become worthless anytime there is any software update.

>My assumption is that OTA updates won't be allowed once this stuff starts requiring certification.

It is unclear whether this would actually be safer or not. I am reminded of how both Tesla[1] and Toyota[2] had similar software problems with their antilock brakes. Both companies had a software fix relatively quickly. Tesla deployed the fix immediately to cars through OTA updates. Toyota issued a voluntary recall meaning its cars wouldn't be updated to the fixed software for months, years, or potentially ever.

[1] - https://money.cnn.com/2018/05/30/technology/consumer-reports...

[2] - http://www.cnn.com/2010/BUSINESS/02/09/japan.prius.recall/in...

[fitzie]

it is a bit scary to think we'd have to lobby government to allow any new invention to prove "safety". Tesla has indicated why it needs to enable these features in order to collect data to improve them, it has also shown that driving under autopilot are already reducing accidents (compared to without and national stats for all vehicles), and yet you call for a entire ban instead of thinking constructively. with the vision stack there will be improvements to the driver attentiveness checks. that would seem to mitigate abuse of the features, which is clearly meant to be supervised at all times by the operator.

[jacquesm]

It's quite simple, really: any software that pretends that it can be in control of a car should be subject to the same kind of test that ordinary drivers have to do before being allowed to take to the road. Then, the manufacturer should assume all liability for errors made by their product. Just like a real person would. They can choose to insure or self insure.

[Gys]

Those successfully tested ordinary drivers also kill many people every year. In 2019 in the US alone more than 35.000!

https://en.wikipedia.org/wiki/Motor_vehicle_fatality_rate_in...

[jacquesm]

Under all circumstances, over all of the miles driven. Tesla is very good at spinning things but in an apples-to-apples comparison they are doing much worse than those ordinary drivers.

[Isinlor]

Where do you have apples-to-apples comparison? As far as I know, this does not exist.

NHTSA is investigating Tesla, some 20-30 accidents. If they find that Tesla is doing something seriously dangerous, I'm sure they will force Tesla to take corrective actions.

The same applies to European regulators.

[jacquesm]

> Where do you have apples-to-apples comparison?

https://www.forbes.com/sites/bradtempleton/2020/07/28/teslas...

[Isinlor]

> But the Autopilot record ballparks to 1.1M miles between accidents off freeway and 3.5M on-freeway.

> By comparison, NHTSA’s most recent data shows that in the United States there is an automobile crash every 479,000 miles.

So, Tesla numbers are better even in the article you link, but still not apples-to-apples comparison. Accident vs. crash.

And Tesla is adding pro forma camera driver monitoring.

[Boritanian]

>where self driving software should be default illegal until proven safe.

Would you apply the same standard to human drivers?

[benhurmarcel]

Yes. It's called a driving license, in many countries it's only delivered after you show you can drive safely.

On a more similar note, that's what we apply to aircraft. They are illegal to fly until proven safe to the authorities (ie. certified).

[papaf]

The same standard is applied to human drivers. I think it is called a "driving test".

[Reason077]

> ”I believe this is a case where self driving software should be default illegal until proven safe.”

You have to remember that human drivers are extremely dangerous, and cause millions of deaths and serious, life-changing injuries worldwide every year.

Any regulations that hold back the development of self-driving software would likely be counter-productive. It’s a bit like arguing that we should have held back Covid-19 vaccines because a small number of people had blood clots or heart inflammation.

[abledon]

no one in government is qualified to really understand ML and to regulate that stuff. Does the government employ people at the level of Tesla in ML? Its quite the problem... when the best minds are in the tech enclaves are not working for the public interest.

[benzor]

We don't need any understanding of AI/ML in government to effectively regulate the autonomous vehicle industry. Design an appropriate set of tests, make companies run the gauntlet, only approve the ones that pass. The test criteria is simple: does this software meaningfully and statistically significantly reduce the risk of accident/harm/death compared to the average human driver? Add caveats and conditionals as you wish for conditions/weather etc. but it's fundamentally a black box test with no knowledge of technical internals required.

[kjksf]

> statistically significantly reduce

And how does one prove something is true "statistically"?

By doing it many times. So many times that you can be statistically confident of the result.

Statistically, there is ~1.4 deaths per 100 MILLION miles driven.

To prove, statistically, that software is as good, it would have to drive at least 1 billion miles. And yes, it would kill 14 people in the process (or more, if it's worse than humans; or less, if its better).

Tesla is already doing what you say car companies should be doing i.e. statistically testing the software

Except you don't like it and think there's a magic fairy test that will show something "statistically" without driving statistically significant number of miles.