[fanf2]

Fuzz testing is incredibly effective at finding gaps in the programmer’s understanding. You should read Barton Miller’s papers on fuzz testing https://pages.cs.wisc.edu/~bart/fuzz/ to see how effective dumb fuzzing still is over 30 years later.

[josephg]

I regularly write custom, small, “fuzzers” in my test suites - at least when I can. For example, recently I’ve implemented a btree for a project with some custom optimisations. My test suite includes a fuzzer which randomly mutates the btree and performs the same mutations on a slower reference data structure. After each change, I test that both structures contain the same data.

The test has shaken out about 10 obscure bugs in my code that my other unit tests failed to find. And that is about what I expected - it’s what I find more or less every time I do randomised testing on code that hasn’t experienced this before.

I really think this sort of testing should be taught and done almost everywhere. It’s wild how many bugs you find with randomly generated input. It is by far the most efficient unit testing you can do, measured by bugs found per line of testing code.

[fanf2]

Definitely, yes!

John Regehr has a nice tutorial, https://blog.regehr.org/archives/896

[r0s]

According to these reports, the effectiveness of fuzzing decreased over that time period.

The majority of bugs reported are explained as poorly designed code, which could then be tested without fuzzing.

For example: A primary class of bugs is unbound inputs, which could easily be found with static analysis. There's no reason to toss random strings at it until it breaks, you can know it will break simply because that input is unbound.

The lack of adequate traditional testing for each utility is specifically mentioned as a limitation of the studies. All fuzzing proves here, is the value of traditional testing. Of course fuzzing is going to find bugs where there are inadequate tests, but there really should be tests.