[loup-vaillant]

You still need your tests to cover all possible errors (or at least all plausible errors). If you try random numbers and your prime happens to be close to a power of two, evenly distributed random numbers won't end up outside the [0,n-1] range you are supposed to validate. Even if your prime is far enough from a power of two, you still won't hit zero by chance (and you need to test zero, because you almost certainly need two separate pieces of code to reject the =0 and >=n cases).

Another example is Poly1305. When you look at the test vectors from RFC 8439, you notice that some are specially crafted to trigger overflows that random tests wouldn't stumble upon.

Thus, I would argue that proper testing requires some domain knowledge. Naive fuzz testing is bloody effective but it's not enough.

[cliftonk]

That’s all true, but fuzz testing is very effective at checking boundary conditions (near 0, near max/mins) and would have caught this particular problem easily.

[loup-vaillant]

Do you mean fuzz testing does not use even distributions? There’s a bias towards extrema, or at least some guarantee to test zero and MAX? I guess that would work.

Also, would you consider the following to be fuzz testing? https://github.com/LoupVaillant/Monocypher/blob/master/tests...

[dwohnitmok]

No, most fuzz testing frameworks I know of these days do not use even distributions. Most use even more sophisticated techniques such as instrumenting the code to detect when state transitions are triggered to try to maximize hitting all code paths in a program instead of repeatedly fuzzing the same path.

[yencabulator]

The usual trick is coverage-guided fuzzing. https://google.github.io/clusterfuzz/reference/coverage-guid...