[pfdietz]

A very funny thing about fuzzing is how random input testing used to be so looked down upon by the software testing community. Read old (1970s) testing books and you'll see comments like "random testing is the worst kind of testing". I still saw this even as recently as a decade ago.

[praptak]

Fuzzing is not random testing though. It's directed random testing.

[pfdietz]

The original fuzzing was as random + black box as it gets.

[UncleMeat]

Yes, and things like coverage guided fuzzing have completely revolutionized things. Prior to directed fuzzing, it was okay but largely unimpressive. Now it blazes through code structures that were previously used as motivating examples for symbolic execution. It is a meaningfully different technique today.

[pfdietz]

Well, that depends on what you mean by "impressive", and in what domain. Black box compiler fuzzing has been very effective.

[UncleMeat]

That's actually one of the few fields where I feel like fuzzing has underperformed. There was an interesting paper at OOPSLA this year that found that while the fuzzing community has indeed found a lot of bugs, that these bugs actually are triggered by real code approximately-never. It was a really interesting result coming from within a community that ordinarily biases towards overinflating the value of PL techniques.

[pfdietz]

That paper, if it's the one I'm thinking of, found that a compiler bug found by fuzzing was more likely to be found by a user, than a user found bug was likely to be found by another user. So if fuzzing-found bug reports are bad, user-found bug reports are even less useful.

Another thing to remember is that as blackbox fuzzing became state of the practice, its benefit declined, as the bugs it would find would be found early, by the developers themselves. All testing techniques are self-limiting this way.

I want you to look at the results of jsfunfuzz and tell me the impact of that wasn't profound.

[m000]

That piece of advice was probably valid in the 1970s: The computers were far too slow and far too expensive for any kind of random testing to make sense. Fuzzing became popular when multi-core CPUs became commonplace and RAM more affordable.

[dnautics]

You really shouldn't do random testing. Fuzzing is better, but property testing (fuzzing with shrinkage) is even better.

[pfdietz]

The original prejudice was against any sort of randomness in testing. Manually constructed tests were seen as superior. That may have been true when computer time was dear, but the bias persisted into the latest edition of a well known book on software testing, published after (for example) Csmith had been released.