[covemarkets]

"Would you want your car or an aircraft you're on piloted by neural net the actions of which can't be explained?" Actually, if the net is properly trained to reduce overfitting and is shown to work on out of sample data - yes. On the flip side of this argument, a person can do something inexplicable and invent a plausible explanation after the fact that seems and feels safe and correct, but is actually wrong.

[TeMPOraL]

Define: "properly trained".

With humans, we generally know the bounds for unexpected behavior. We understand tiredness, confusion, fear, distraction, suicidal thoughts and other factors. We also know how to screen people to minimize those bounds.

With ML stacks, we have no good grasp on bounds. They usually work, for some definition of working, up until they don't - and when they fail, it's in some absurd (therefore hard to predict) way.

[covemarkets]

humans also fail for all sorts of reasons that are unpredictable. if, for example, Waymo, demonstrates properly calculated lower accident rates with their cars than with humans, then at some point you have to agree with what the data says. We aren't there yet, but that doesn't mean we can't get there

[TeMPOraL]

If they demonstrate, sure.

But for that to be useful they also need to demonstrate that their model is robust under modifications - otherwise every time they retrain their NNs they should throw away prior safety records and start counting from zero. Because at this point, what would be the argument to keep it? With humans we know - from thousands of years of experience - that people generally don't go crazy when taught new things. With NNs, we know they're very sensitive to training data.

[covemarkets]

Well, you are making pretty general statements. "With humans we know - from thousands of years of experience - that people generally don't go crazy when taught new things"

they do crazy things all the time. Drink and drive, hijack planes and run them into mountains ( germany, malaysia recently ), text and drive, etc. etc. This discussion needs to be based on comparing actual data. Let's get there and see

[TeMPOraL]

This is one topic when you can sort of get away with general statements, because it's human interop - we all know how it works :).

I covered drunk driving and suicides in my original comment. This happens, but we know it does, we know how often and why it does, and know how to work around it.

What I was thinking about wrt. learning failure modes is this: when you put a person through 30+ hours driving course, they don't suddenly lose the ability to recognize trees or faces. The same cannot be said about retraining existing neural networks.

I'd love for the actual data to appear for analysis. Right now I'm worried about the very concept of using a black-box bag of statistical tricks DNNs are for safety-critical operations. How is it that we can't handle the problem of self-driving with more direct, stable and auditable methods?

[covemarkets]

I think you are putting a bit too much faith in human cognition: "when you put a person through 30+ hours driving course, they don't suddenly lose the ability to recognize trees or faces". They do. they get tired and their reaction time slows down. they get drunk, they get distracted. Humans do exactly this - unpredictably lose the ability to recognize and react in a timely manner.