Hacker News new | ask | show | jobs
by nuclx 3130 days ago
As proposed by Dan Luu: https://danluu.com/testing/
1 comments

As I explain in the RamFuzz overview paper [0], Dan Luu's idea is unlikely to work because fuzzing uses mutation techniques developed for file input, not individual parameter values. What does seem to work is frameworks like EvoSuite and eToc, which mutate method-invocation sequences to try and maximize coverage.

And yes, QuickCheck (or RapidCheck[1], for C++) is what you need if you enjoy manually specifying your test inputs. RamFuzz is a tool to support research into automating that and, eventually, the entire process of writing unit tests.

[0] https://github.com/dekimir/RamFuzz/blob/master/sci/ramfuzz.m...

[1] https://github.com/emil-e/rapidcheck

It doesn't sound like you're talking about the same thing. Dan Luu's idea sounded more about adding coverage guidance to QuickCheck, which presently lacks it, not using fuzzing-like mutators.
But how exactly does Dan propose to add coverage "guidance"? Here's what he says:

"Since Zalewski has already done the work of figuring out, empirically, what heuristics are likely to exercise more code paths, it seems like a waste to ignore that and just generate totally random values."

Dan does, in fact, explicitly suggest "using fuzzing-like mutators", as you put it, in QuickCheck's random value generator. But those mutators work on continuous file contents, not on a bunch of disparate parameter values generated at distinct moments in the program. An early version of RamFuzz tried to apply AFL to the log file recording all generated random values, but all that yielded was invalid logs that couldn't be replayed. I don't know how else one would leverage Zalewski's heuristics in a random-value generator, though I'm obviously open to suggestions.