[slg]

>I believe this is a case where self driving software should be default illegal until proven safe.

If we go with your suggestion, what would you consider "proven safe"?

Autopilot has been running in hundreds of thousands of cars, has driven several billion miles, and we still don't have enough data to prove whether it is safer or more dangerous then an average human driver. Accidents are rare and we therefore need a huge amount of data before we can be confident that the accident rates we are seeing are predictive. I have no idea how you collect that type of data without them being tested on real roads with other real drivers.

[dragontamer]

How about actually passing some basic tests before they're deployed?

(Warning: unnecessarily loud video for some reason): https://twitter.com/finance_degen/status/1307529357951467531

Lets start with passing these basic tests before selling them as working features. I don't think we're asking for a very high bar.

[slg]

There is zero context on that video that tells us what is happening there. We have no idea if that is the Autopilot failing or the emergency braking failing. We also have no control group to tell us what percentage of humans would stop short of that dummy. It is inexact due to Twitter's video player not showing fractions of a second, but it looks like there were approximately 2 seconds between when the dummy started moving forward and when it was hit by the car. The average human time to braking is 2.2-2.3 seconds[1]. Is the car even failing that test in comparison to a human?

It also isn't clear from watching that video what the safest and therefore desired behavior should be in that situation. A self driving car is obviously not going to prevent all accidents, so it is a question of minimizing potential harm. We don't want a car to aggressively brake whenever someone at a street corner takes a step towards the road. We therefore need to balance the chance of a person stepping into the path of the car with the risk of braking when it is unnecessary and causing a rear end collision. The problem in the linked thread is overaggressive braking so forcing the car to pass a test that rewards overaggressive braking would only make that specific example worse.

That leads back to my point about needing a huge amount of data. You can't just run a car through an obstacle course to know whether it is safer or more dangerous than a human. You need to have it interacting with unpredictable humans and you need to do it repeatedly before you can confidentially predict whether it is safer or more dangerous than a human.

[1] - https://copradar.com/redlight/factors/IEA2000_ABS51.pdf

[dragontamer]

> We also have no control group to tell us what percentage of humans would stop short of that dummy.

IIRC, Subaru and other companies pass these simple emergency braking tests 100% of the time.

That test was a Chinese test IIRC, but the software doesn't change between countries. Similar tests have been done here in the USA by insurance groups to set insurance rates, but a government-mandated test for what "emergency braking" really means (before you "sell the feature to the public", lets actually have a government-mandated test similar to that video).

You shouldn't be allowed to call your stuff "autopilot" or "full self driving", or "emergency braking" or "pedestrian avoidance" (or some other set of words) unless you can... you know, avoid pedestrians and emergency brake in a well-controlled test.

Avoiding balloon people is enough. But its a well known fact that Tesla repeatedly fails at these simple tests, when other groups (ie: Mobileye group / Mobileye hardware) manages to emergency brake in time.

----------

IIHS test: https://www.iihs.org/news/detail/performance-of-pedestrian-c...

The issue is that 3rd party non-government groups (ie: IIHS) are the ones running these tests. There's no advocacy group for US consumers as far as I can tell. IIHS is primarily about serving their master (insurance companies).

Don't get me wrong: IIHS is doing good work here. But its not their job to protect the consumer.

EDIT: I got my sources mixed up. Tesla apparently passed the IIHS test.

It was the AAA test they failed: https://insideevs.com/news/377427/video-tesla-model-3-failed...

[slg]

>IIRC, Subaru and other companies pass these simple emergency braking tests 100% of the time.

They do not pass 100% of the time unless you have a very narrow definition of "these simple emergency braking tests". No emergency braking system is foolproof.

>But its a well known fact that Tesla repeatedly fails at these simple tests,

You say this while at the same time the source you include has Tesla in the middle tier of results.

Either way, my point is not that Teslas are safe or that they perform well on this test. The point is that this test does not tell you whether a car being driven by Autopilot is safer than a human.

[dragontamer]

> They do not pass 100% of the time unless you have a very narrow definition of "these simple emergency braking tests"

Lets get them working consistently under well defined, standard, simple, emergency braking tests before worrying about the real world.

Like not hitting a balloon dressed up as a pedestrian during clear skies in sunny weather. I don't care about rainy days until we get the bright / sunny weather figured out.

[slg]

>Lets get them working consistently under well defined, standard, simple, emergency braking tests before worrying about the real world.

Automatic emergency braking is the exact wrong feature to use for your example. Either the driver sees the pedestrian, stops in time, and the automatic emergency braking is of no use or the driver would have hit the pedestrian and any effort from the automatic system is a benefit. This is the type of feature that should be deployed as soon as possible assuming it is not tuned too aggressively to stop at false alarms.

[HWR_14]

> we still don't have enough data to prove whether it is safer or more dangerous then an average human driver.

I would point out that given the dangers involved with accidentally turning over a million vehicles into autonomous 5000 lbs missile, erroring on the side of caution seems fine. The benefits are quite low: if the autopilot had been on since inception between 20 and 100 lives would have been saved (I accept your "several billion miles" number and point out that that the average fatality rate is 1.1 per 100 million miles driven, but that is based on averaging in 40 year old cars with fewer safety features and shrinks ever year.). The costs could be astronomical: a simultaneous failure (security, mistraining, date bug, whatever) could result in hundreds of thousands or millions of deaths.

Which means there are three errors to consider. (1) Obviously, some things (bugs, exploits) are unknowns and there will always be an inherit risk there. I would say that these risks may forever make self-driving cars too risky. (2) It is difficult to come up with any actual test of driving skills. This is especially true because any test will suddenly become the target so we have to have the test cover everything. (3) Actual driving errors: Both of the above assume that the AI can drive as well as a person. That's obviously difficult to do. And we would need to see a huge improvement to justify adding a new risk factor.

[slg]

>The benefits are quite low: if the autopilot had been on since inception between 20 and 100 lives would have been saved

This is only the case if you look at the current system as the finished product. The biggest benefit is that it gets us closer to a true self driving system. That would not only save millions of lives, but it would revolutionize logistics and economics of transportation which can in turn reshape society.

>The costs could be astronomical: a simultaneous failure (security, mistraining, date bug, whatever) could result in hundreds of thousands or millions of deaths.

I have no idea what scenarios you are imaging that could lead to "hundreds of thousands or millions of deaths." Almost every Autopilot death in the US makes national news. There is no way hundreds of people could die without there being some type of intervention in the system.

[northwest65]

> Autopilot has been running in hundreds of thousands of cars, has driven several billion miles,

Has it really though?

> I have noticed for me at least it started happening after I updated at 2021.4.18.11

This implies that the functionality of the AutoPilot is constantly changing, presumably meaning each version has thousands of miles rather than AutoPilot having 'several billion miles'. It doesn't seem like you can trust past performance is the users are to be believed.

My assumption is that OTA updates won't be allowed once this stuff starts requiring certification.

[slg]

I think the first half of your comment is a pointless semantic debate. The Autopilot system has driven billions of miles. Those miles obviously all aren't equally relevant. The older miles lose value as the hardware or software changes. However those miles don't all become worthless anytime there is any software update.

>My assumption is that OTA updates won't be allowed once this stuff starts requiring certification.

It is unclear whether this would actually be safer or not. I am reminded of how both Tesla[1] and Toyota[2] had similar software problems with their antilock brakes. Both companies had a software fix relatively quickly. Tesla deployed the fix immediately to cars through OTA updates. Toyota issued a voluntary recall meaning its cars wouldn't be updated to the fixed software for months, years, or potentially ever.

[1] - https://money.cnn.com/2018/05/30/technology/consumer-reports...

[2] - http://www.cnn.com/2010/BUSINESS/02/09/japan.prius.recall/in...