|
|
|
|
|
|
by pfdietz
1909 days ago
|
|
|
Hypothesis does shrinking in an interesting way. The first idea you think of for shrinking it to take the randomly generated values and try to make them smaller. But the generator may be imposing constraints on the values, and if you lose those constraints, the input becomes invalid. An example of this problem is generating valid C programs to test C compilers (by compiling with different compilers or different optimization settings and seeing if the behavior differs). The constraint there is that the C program not show undefined or implementation-specific behavior. Naively shrinking a C program will not in general preserve this property. Hypothesis takes a different approach to shrinking: it records the sequence of random values used by the generator, and replays the generator on mutations of that sequence that do not increase its length. The only way this can fail is if the generator runs out of values on the mutated sequence. Otherwise, the new output will always satisfy the constraints imposed by the generator. Hypothesis does various clever things to speed this up. |
|
|
Eg the simplest example might be an application where you input an integer, but a smaller int actually drives up the complexity. This idea gets more complex when we consider a list of ints, where a larger list and larger numbers are simpler. Etcetc.
It would be neat if a proptest framework supported (and maybe they do, again, i'm a novice here) a way to rank complexity. Eg a simple function which gives two input failures and you can choose which is the simpler.
Another really neat way to do that might be to actually compute the path complexity as the program runs based on the count of CPU instructions or something. This wouldn't always be a parallel, but would often be the right default i imagine.
Either way in my limited proptest experience, i've found it a bit difficult to design the tests to test what you actually want, but very helpful once you establish that.
[0]: https://crates.io/crates/proptest