[seren]

I have a hard time understanding the current AV SW stack.

On one hand, at the low level, sensor, motor control, etc you likely have traditional hard real time/MISRA C code, but on the higher layers you probably things like DNN, image recognition, which are much less deterministic.

So I am not sure how do you reconcile these two worlds, and prove it is safe and always work in timely manner.

It seems the only sound approach would be to validate the whole system on a real road.

[yoav_hollander]

A few comments on this:

First, as etendue says, it is not easy. The problem of mixing “Boolean” verification with probabilistic, less-deterministic verification is especially hard. I discussed this a bit in [1], if you care to take a look.

Also, I think most current AVs are not driven by DNNs at the top level (comma.ai [2] is one exception). See [3] for some discussion of that, and of verifying machine-learning-based systems.

Finally, one possible way to check that AV manufacturers “do the right thing” in correctly verifying the combination of DNNs, Misra C, digital HW, sensors and so on is perhaps to create a big, extensible catalog of AV-related scenarios, which ideally should be shared between the manufacturers and the certifying bodies – see [4]. I think there is some hint of that in the DOT pdf – still working my way through it.

[1] https://blog.foretellix.com/2016/07/22/checking-probabilisti...

[2] http://www.bloomberg.com/features/2015-george-hotz-self-driv...

[3] https://blog.foretellix.com/2016/09/14/using-machine-learnin...

[4] https://blog.foretellix.com/2016/07/05/the-tesla-crash-tsuna...

[seren]

Thanks for your input, really interesting topics on your blog as well.

[yoav_hollander]

Thanks. I did a second pass through the policy paper, and put a summary of the verification implications here: https://blog.foretellix.com/2016/09/21/verification-implicat...

[etendue]

I think the simple answer is that it is not easy. To start, rigorous design processes with risk analysis upfront are certainly necessary, as are well-defined operational contexts for the autonomous functionality, and a very disciplined approach to clearly defining safety-critical subsystems and minimizing their surface area.

There's a surprising amount of work in the literature that serves as a guide for using neural networks in safety-critical contexts, e.g., http://dl.acm.org/citation.cfm?id=2156661 and http://dl.acm.org/citation.cfm?id=582141.

[rubidium]

Now you understand the job of systems engineering :)

Verify components, validate the entire system is the typical approach.

[DougWebb]

That sounds pretty much just like web application development, or any other front-end user-facing development. You can verify internal components through testing, but once you introduce non-deterministic random variables like browser software your user is using, and your user's themselves, all you can do is validate the entire system through real-world testing and hope you've covered the edge cases you need to handle and will fail gracefully for the ones you missed.

[seren]

The point I was trying to make is that if you have actuators running MISRA C that are going to be driven by something written in Tensorflow, does it still makes sense to have a requirement to use MISRA C in the first place for the low level part ?

[etendue]

I'd be very wary of using complex SOUP like TensorFlow, even if brought under my quality system. I think a good answer here is that once one goes under design control the subset of functionality needed should be implemented in-house under the organization's SDLC.

[ansgri]

Of course these things are meant to be used (1) to train the system, (2) as a player in the prototype. Exactly like in the old school ML-based systems: you train in Matlab or CudaConvNet, and then you load the trained classifier into the custom-made player highly tuned to your hardware and problem domain.

[Qworg]

Most certainly - safety should be guaranteed at the lowest level, even if Tensorflow gets borked.

Think of it as a failure cascade - if Tensorflow breaks, the car can safely stop. If the low level stuff breaks, the car may not be able to stop (or go).