[hinkley]

There's a phenomenon I've witnessed in dev teams that I always thought was a separate issue but I'm starting to question that conclusion.

Like with Covid, like with SLAs, it's not enough to think of ratios of success or failure. You have to also look at frequency and duration.

A flaky test might fail a build 1 time in 200, but as your team gets bigger, build frequency rises, your tests grow, and eventually you're getting failed builds frequently enough that people start to see them as a regular occurrence, and that negatively affects their opinions about the whole experience. I've seen people bash the system when failures happen weekly, I've seen others 'turn' after a couple of statistical clusters and then fall to confirmation bias long afterward.

Google has so many irons in the fire that I think we've reached that same threshold for a lot of people. Shutting down the worst 1% of your projects a year sounds like a completely reasonable business plan. Until you have 1000 projects, and now you're shutting one down every five weeks on average. People will talk.

And if there's no transparency in that process, how do I know that my favorite tool isn't next, or on the list for next year? Odds are low, but not zero.