Hacker News new | ask | show | jobs
by kybernetikos 2108 days ago
This article makes me think that we should consider tests as a monte-carlo integration over the space that verification tries to cover.
1 comments

Interesting perspective. It only really works with property based testing, not with conventional fixed-example based testing.
It is definitely more natural with property based testing, but I think it works to some extent as a perspective in general.

You have a huge input space where each point represents some combination of possible inputs, you can think of representing correct behaviour as 0 and bad behaviour as 1 for each of those points. You're attempting to show by your verifying and testing as best as you can that the sum across all the inputs is 0.

For fixed example testing, you sample some points that you consider to be representative or boundaries, show that they are 0, and use that to argue that the whole space is 0. With property testing, you can let it literally do the monte-carlo thing. With verification you attempt to do the integration analytically.

All good points!

I think just as reliability engineering benefited moving from 'keep the system always up' (but fail occasionally in the real world) to 'keep the system up 99.999% of the time' (and succeed in a measurable way), fixing bugs in software might benefit from such an honest and quantitative approach.