Hacker News new | ask | show | jobs
by lukego 2576 days ago
I have learned to love non-deterministic tests.

The world is non-deterministic. A test suite that can represent non-determinism is much more powerful than one that cannot. To paraphrase Dijkstra, "Determinism is just a special case of non-determinism, and not a very interesting one at that."

If a test is non-deterministic then a test framework needs to characterize the distribution of results for that test. For example "Branch A fails 11% (+/- 2%) of the time and Branch B fails 64% (+/- 2%) of the time." Once you are able to measure non-determinism then you can also effectively optimize it away, and you start looking for ways to introduce more of it into your test suites e.g. to run each test on a random CPU/distro/kernel.
2 comments
muro 2576 days ago
But you pay the cost of retrying the failing tests and lack of clear signal. And if the application code is flaky, users get to experience the breakage too.
link
lukego 2576 days ago
If an application is flaky then I want to know: How frequently does it fail? How does this depend on combinations of configuration parameters? How does this compare between the stable, master, and next branches? etc.

The best way that I know for doing this is to write tests that are flaky because they expose the underlying flakiness in the application.

If an application is flaky and its test suite always runs 100% then I'd be pretty suspicious about that test suite being adequate.

link
mrkeen 2576 days ago
> And if the application code is flaky

This is the only relevant factor. Forget the rest. Users don't experience your flaky tests just like they don't experience your messy Jira boards or your bad office coffee.

link
AstralStorm 2576 days ago
How do you know which is failing without exhaustive analysis?

See, once you know why the test fails and it's not the tested application, which is exceedingly rare in practice, you can just disable it or fix it. But only if you're actually sure, not before.

link
muro 2575 days ago
In my experience it is usually the test.
link
lukego 2575 days ago
In my experience that's because tests are usually written in a timid style that tries not to provoke flaky behaviour from applications.

If your test suite can handle non-determinism then you can approach test-writing in a completely different - braver - way.

link
mrkeen 2576 days ago
Yes! To paraphrase John Hughes, "Every time you run your test suite, you should become more confident in your software."
link