[tybug]

(not OP but I would be surprised if the answer wasn't) yes, because you're changing the order in which the random draws are interpreted. But this isn't a problem in practice because you generally aren't changing the generator in the middle of debugging a failure.

[parentheses]

So, this means refactoring becomes potentially difficult. While the gem is still a great accomplishment and very useful, I'd have to engineer my way around this issue before using it with things like a Rails Model which could have changing shape.

@OP:

I wonder if the README (and possibly runner) should suggest writing a test-case that doesn't rely on PBT when the user wants to preserve a case for future testing.

The issue here is that if you're saving a singular example and it represents a weird corner case, it's totally conceivable that a small change will result in an invisible change to that test case.

Another idea: it'd be great if the test could simply take examples that are failing and add them to a `failing_examples.rb` or some such. I know I'd use a feature like this quite a bit.

[ohbarye]

I think there are usually three actions a programmer can take when PBT fails.

- Create a test case that doesn't depend on PBT as you suggest.

- Fix the production code being tested since its failure is an unexpected bug.

- Fix the PBT itself. This means that the programmer has had wrong assumption for the test target.

I think it's difficult for the tool to know which choice is the best on a failure. But if there's any good idea, I'd like to incorporate it. :)

[ohbarye]

>yes, because you're changing the order in which the random draws are interpreted. But this isn't a problem in practice because you generally aren't changing the generator in the middle of debugging a failure.

Correct. The test inputs are determined by a seed and generators (including the order of generators).