[r0s]

Fuzzing is guesswork. I see this attitude: because an unknown bug could possibly exist, therefore it could be a high severity bug and is then worthy of expending a lot of time and effort to discover.

To beat all odds and discover something you can't predict, you don't even know what it could be. Some effort should be done to reduce the problem space.

In the given example, I don't see why you need to test input at the cli level when access control and input sanitation should be verified already using known parameters that reject all unpredictable input. Obviously, certain combinations of input are more dangerous than others, and at the very least, those individual systems should have focused parameterized tests, and that set reduced from the random fuzz possibilities.

Exploratory testing on highly secure, safety prioritized systems is one thing. Sure, chaotic testing like this has a place, in a very specific, hopefully highly structured system. Even then I would use it only after every other type of testing.

When someone wants to test every input possibility with random noise I roll my eyes. Test what you know is a threat first, achieve solid coverage, run tests at every stage of development and then maybe we can talk about fuzzing. Is the system actually functioning as it's intended? Are all the happy path use cases tested? Are you sure about that? Boring, I know.

[fanf2]

Fuzz testing is incredibly effective at finding gaps in the programmer’s understanding. You should read Barton Miller’s papers on fuzz testing https://pages.cs.wisc.edu/~bart/fuzz/ to see how effective dumb fuzzing still is over 30 years later.

[josephg]

I regularly write custom, small, “fuzzers” in my test suites - at least when I can. For example, recently I’ve implemented a btree for a project with some custom optimisations. My test suite includes a fuzzer which randomly mutates the btree and performs the same mutations on a slower reference data structure. After each change, I test that both structures contain the same data.

The test has shaken out about 10 obscure bugs in my code that my other unit tests failed to find. And that is about what I expected - it’s what I find more or less every time I do randomised testing on code that hasn’t experienced this before.

I really think this sort of testing should be taught and done almost everywhere. It’s wild how many bugs you find with randomly generated input. It is by far the most efficient unit testing you can do, measured by bugs found per line of testing code.

[fanf2]

Definitely, yes!

John Regehr has a nice tutorial, https://blog.regehr.org/archives/896

[r0s]

According to these reports, the effectiveness of fuzzing decreased over that time period.

The majority of bugs reported are explained as poorly designed code, which could then be tested without fuzzing.

For example: A primary class of bugs is unbound inputs, which could easily be found with static analysis. There's no reason to toss random strings at it until it breaks, you can know it will break simply because that input is unbound.

The lack of adequate traditional testing for each utility is specifically mentioned as a limitation of the studies. All fuzzing proves here, is the value of traditional testing. Of course fuzzing is going to find bugs where there are inadequate tests, but there really should be tests.