[fluffything]

> What about input data?

What about it? You get input data from data sources which, e.g., in a car it would be a sensor. The manufacturer of the sensor provides you with the guaranteed sensor accuracy for some inputs, which gives you the upper bound on the input error from that source.

That is, in a reliable control pipeline, the upper bounds on the errors of data sources are known a priori.

Sure, sensors can malfunction, but that's a different problem that's solved differently (e.g. via resiliency using multiple sensors).

> A control system doesn't need to be end-to-end neural, by the way.

Who's talking about end-to-end neural nets for control? If a single part of your control pipeline has unknown error bounds, your whole control system has unknown error bounds. That is, it suffices for your control pipeline to use a NN somewhere for it to become unreliable.

This doesn't mean that you can't use control systems with unknown error bounds somewhere in your product, but it does mean that you can't trust those control systems. This is why drivers still need to keep their hands on the steering wheel on a Tesla: the parts of the pipeline doing the autonomous driving use NNs for image recognition, and the errors on that are unknown.

This is also why all "self driving" cars have simpler data sources like ultrasonic sensors, radar, lidar, etc. which can be processed without NNs to avoid collisions reliably. You might still use NNs to improve the experience but those NNs are going to be overridden by reliable control pipelines when required.

[red75prime]

> That is, in a reliable control pipeline, the upper bounds on the errors of data sources are known a priori.

Now we have natural neural networks in the control loop for some reason despite their unknown error bounds. To give another example: is there a sensor for vehicle placement relative to road edges with known upper error bound, which is less than width of the road? No, we have GPS, radar, lidar, camera data that we need to interpret somehow.

A car that reliably avoids collisions (Can it, though? It needs to predict road situation to do it reliably), but can occasionally veer off the road, doesn't strike me as particularly safe.

> to be overridden by reliable control pipelines when required.

Those reliable pipelines needs to be mostly reactive. And there's a limit on what they can do. You can't avoid a collision when a car emerges from around a corner with 0.1 seconds to react. You need complex processing of those "simple" data sources to detect zones that can't be observed right now and to assess a probability of the said situation.

All in all, we already have unreliable human part in the control loop of a vehicle. A control system that is provably robust in all real world situations will be the ultimate achievement and not a prerequirement for wide use of self-driving cars.

[fluffything]

> Now we have natural neural networks in the control loop for some reason despite their unknown error bounds.

Not in any automatic control loops. All control loops that do this have a human as the final piece of the pipeline, and that human is legally responsible for the outcome of the control loop. That defeats the point of automatic control.

> To give another example: is there a sensor for vehicle placement relative to road edges with known upper error bound, which is less than width of the road?

No, which is why these control pieplines have a human in control. For the experimental pipelines that do not have a human in the end of the control loop, they do have other control pipelines to avoid collisions, and the only that their control algorithms guarantee is a lack of collisions, not the ability for the car to stay on a lane. That is, the car might leave the lane under some conditions, but if it does, it will detect other objects and avoid crashing into them (although those objects might crash into it).

> All in all, we already have unreliable human part in the control loop of a vehicle. A control system that is provably robust in all real world situations will be the ultimate achievement and not a prerequirement for wide use of self-driving cars.

Right now, control loops without proven error bounds are not allowed by certification bodies on any control-loop in charge of preserving human lives in the aerospace, automotive, medical, and industrial robotics industries.

Allowing control-loops without known error bounds to be in charge of human lives would lower the current standards of these industries. Could be done (the government would need to create a new kind of regulation for this), but at this point it is unclear how that would look like.

[sudosysgen]

Well actually, they are allowed in the automotive world, re: Tesla.

They won't necessarily lower the standard for some applications if the alternative is just regular people at the wheel.

[fluffything]

> Well actually, they are allowed in the automotive world, re: Tesla.

This is false: Tesla's "autopilot" isn't a control-loop, much less a certified control-loop. What Tesla calls "autopilot" is actually a "driving assistant". It is not in charge of controlling the car, but it is allowed to assist the driver to control the car.

This is a subtle but very important difference, since this is the reason that Tesla tells its owners to always keep the hands on the steering wheel, and Tesla cars are required to disable their "driving assistant" in countries like Germany if the driver hands leave the steering wheel. It is also the reason that Tesla is not liable if a car with "autopilot" kills somebody, because the "autopilot" isn't technically driving the car, the driver is.

The word "autopilot" comes from the aerospace industry, where pilots are not required to keep their hand on the controls or pay attention when the "autopilot" is on, and the manufacturer is liable if the "autopilot" screws up (e.g. see Boing). Tesla's usage of the word "autopilot" to refer to driving assistants is misleading and dangerous.

A true car "autopilot" is what people call "Level 5" in the autonomous driving community. Elon Musk promised to ship 10.000 self-driving Level 5 Tesla taxis in 2019, and now mid 2020. We'll see about that.Elon Musk has been saying that Level 5 will happen next year for the last 10 years, so at least when it comes to autonomous driving, their predictions have been consistently wrong.

Most CEOs who adventure to predict when Level 5 will happen say something like "not before 2030". And Waymo's goal is to achieve """Level 5""" in a very small restricted area of Phoenix downtown for some restricted weather conditions some time between 2020-2030, but it is unclear what the road after achieving it would be for certification, and the road from there to actual, real, Level 5 is still unclear at this point.