[hooloovoo_zoo]

I haven't seen a paper that shows this "tricking" can be used as a real world attack or happen randomly. Just because you can compute an input that has this unusual behavior doesn't mean there is a demonstrably nonzero probability of it happening.

[dkarapetyan]

Wait? That's exactly what it means. Since the networks are not "continuous" you can't reason about how the system will behave in actual real world conditions because any random fluctuations can cause the whole thing to malfunction. I put continuous in quotes because it's not the real definition of continuous like in real analysis but a good enough analogy as in small variations in input should not lead to wildly different outputs.

This is why any model that lacks explanatory power can't be used in mission and safety critical systems. If it can't reason about things the same way people can reason about things then the system overall can't really be trusted. It's one thing when a translation from english to spanish is wrong, it's a completely another thing when the control software of a self-driving car decides to accelerate instead of break and the root cause analysis is people throwing their hands up and saying neural networks are inherently susceptible to these kinds of problems.

[argonaut]

To be fair, you should be more precise. The attacks are specifically calculated. The combinatorial space of possible inputs is so massive that I'm sure it is extremely unlikely for a malicious input to occur randomly.

[dkarapetyan]

I don't think it has to do with the combinatorics of the input space. Adversarial inputs are hard to generate until someone figures how to point a set of laser pointers at exactly the right spots on a truck on a highway to get it to swerve out of control.

[sgk284]

How is that different than today with human drivers? A laser to the eye will cause lots of swerving.

[argonaut]

> because any random fluctuations can cause the whole thing to malfunction

You made a very specific claim that random fluctuations could have the same effect as adversarial examples. I was addressing that.

[dkarapetyan]

Yes, that makes sense and you're right. I don't have a proper definition of randomness and I wouldn't expect generic noise to cause issues.

[hooloovoo_zoo]

These changes are not random. The whole reason neural networks work at all is because probable differences do not mislead the network.

[dkarapetyan]

This is not true at all. Adversarial input can indeed be probable input depending on your definitions and I haven't seen anything yet that describes the probability distributions of inputs. Everyone takes a bunch of training examples and extrapolates from there.