[UncleMeat]

Yes, and things like coverage guided fuzzing have completely revolutionized things. Prior to directed fuzzing, it was okay but largely unimpressive. Now it blazes through code structures that were previously used as motivating examples for symbolic execution. It is a meaningfully different technique today.

[pfdietz]

Well, that depends on what you mean by "impressive", and in what domain. Black box compiler fuzzing has been very effective.

[UncleMeat]

That's actually one of the few fields where I feel like fuzzing has underperformed. There was an interesting paper at OOPSLA this year that found that while the fuzzing community has indeed found a lot of bugs, that these bugs actually are triggered by real code approximately-never. It was a really interesting result coming from within a community that ordinarily biases towards overinflating the value of PL techniques.

[pfdietz]

That paper, if it's the one I'm thinking of, found that a compiler bug found by fuzzing was more likely to be found by a user, than a user found bug was likely to be found by another user. So if fuzzing-found bug reports are bad, user-found bug reports are even less useful.

Another thing to remember is that as blackbox fuzzing became state of the practice, its benefit declined, as the bugs it would find would be found early, by the developers themselves. All testing techniques are self-limiting this way.

I want you to look at the results of jsfunfuzz and tell me the impact of that wasn't profound.