[spockz]

I would like to make a shout out to quickcheck (Haskell, https://hackage.haskell.org/package/QuickCheck) as it was my first introduction to property based testing and has since let me to use property based testing everywhere in the last 10 years. Then there is scalacheck (https://www.scalacheck.org/). Both let you write your own generators and are quite good at shrinking down to minimal use cases.

The suggestion elsewhere in this thread to decrease the number of iterations during normal testing and crank it up during nightlies is also good.

The only thing I’m still missing from the libraries is a convenient mechanism of remembering previously failing generated inputs and use them as a static list of testcases next to the at runtime generated ones like a regression test of sorts.

Edit: typos

[dwohnitmok]

> I would like to make a shout out to quickcheck (Haskell, https://hackage.haskell.org/package/QuickCheck) as it was my first introduction to property based testing

As far as I know the Haskell Quickcheck library by John Hughes was in fact the first library to lay out property-based testing. He then went on to create a paid and expanded version of the library in Erlang. And then as Quickcheck rose in popularity it's been re-implemented in many many different languages.

[im_down_w_otp]

Indeed. It's an unfortunate general omission in the ecosystem. It's one of the things that our product does (in its appropriation from different inspirations like MC, PBT, SBFL, etc., but we move things up to assessing systems rather than programs) where contradictions and counter-examples are curated specifically for making it easier to reuse them. They're provided back to the user as executable "properties" in our DSL to apply to past, present, future data.

disclaimer: Auxon co-founder

Also, big (h/t) to Quickcheck from me as well. Getting into it via Erlang many, many years ago was among the more impactful and transformative developments in my approach to thinking about improving software quality.

[spockz]

In the article you (auxon) wrote to avoid using type based generators and unbounded collections and instead write your own generators. Usually I find it more convenient to write filters on the generated types then create my own generators. As the post mentions, this can lead to many discarded inputs. Have you ever considered how we could use the predicates in the filters to create Specialized generators that only generate inputs that will be accepted by the filter? You probably need some meta programming or reification of the predicates for that to work.

[mristin]

You can already do that with a family of predicates if you write preconditions in your Python code (see my previous comment [1]). There is an ongoing discussion how to bring this in into Hypothesis (see the issue [2]).

[1]: https://news.ycombinator.com/item?id=26018386

[2]: https://github.com/HypothesisWorks/hypothesis/issues/2701

edit: newlines

[mullr]

> Have you ever considered how we could use the predicates in the filters to create Specialized generators that only generate inputs that will be accepted by the filter? You probably need some meta programming or reification of the predicates for that to work.

That's a really nice idea. It doesn't fit into the usual compositional design of the proptest libraries that I've used, but it certainly seems like it should be possible in principle.

[amw-zero]

Hypothesis does what you’re talking about. It stores failing examples and chooses them during subsequent runs.