[pfix]

You measure how many of your deployments ship broken code. I've worked in contexts where test coverage was sufficient enough to be confident that when deploying there wouldn't be any issues apart from really novel unexpected stuff you wouldn't have tested anyway.

I've also worked in contexts wher I didn't have that confidence.

I would say you know when you're in that environment and directly measuring doesn't really help as it's more a matter of culture than technical metrics.

[pfix]

Because it's linked in another post right now:

If you feel confident that renovate[1] pull requests can be merged without review on a green pipeline, you have 100% confidence in your tests ;-)

My first context was that way.

[1]: https://www.jvt.me/posts/2024/04/12/use-renovate/

[mewpmewp2]

What do you mean by "broken code"? Any sort of code that yields in a bug? Any sort of bug? Because interpreting directly broken code implies to me some sort of compilation error. Also measuring that seems very complicated, depending on how you interpret "broken code". E.g. a bug could influence 100% of the customers, 5% of the customers, 0.1% of the customers and only when certain conditions happen, it could be a combination of deployments introducing this. It could be a bug that can easily be worked around by a customer. It could be a bug that only arrives 6 months down the line, because some dependency changed, and you didn't have proper handling for it, even though it was expected that you had. E.g. you are calling a third party API that alerts you in advance they are planning to add extra error code in the system in 6 months, so you should probably also include it, but you don't and then system bugs due to that.

The problem could arrive 2 years in the future when scale increases and the code wasn't built to handle it, even though it was expected.

You take 100 of your past deployments and see how many resulted in some sort of a bug? E.g. if there's 0, then you have 100% confidence?

> any issues apart from really novel unexpected stuff you wouldn't have tested anyway.

I mean that's one thing that makes it impossible to have it 100% for me at least.

> I would say you know when you're in that environment and directly measuring doesn't really help as it's more a matter of culture than technical metrics.

I may have been in wrong environments throughout my whole life, but I also have trouble imagining a perfect environment where there's 100% confidence.