[Scubabear68]

The article starts with “Last year, we made the difficult decision to stop deploying any changes to production for one week” and goes on to talk about releases.

In that context I assume this means they make multiple production releases per day (which makes me shudder). I am curious how they do this while maintaining high quality and not driving customers insane.

[sdairs]

Hey, Al from Tinybird here (co-author of the post). We've made up to 20 production releases per day some days. It's transparent to our users, they aren't even aware the upgrade is happening, there's no upgrade button to hit, there's no downtime. We release often because we release small and fast. It's not like those 20 releases are always fundamentally changing the product. We would rather fix a minor bug or two and get that out to our customers ASAP, than hold on to it for a few months and drop a huge change. In a vast majority of cases, a user won't even consciously notice something changed.

Doing this kind of fast iteration has its risks, but it has its benefits too. We de-risk it, in part, by having extensive CI, which is why it was so important to us that the CI is fast & reliable.

Delivering larger, less-frequent updates has its own risks. You're not practising your release process as frequently, so it's a much bigger event. You're pushing many, many more changes in one go, so there's a lot more surface area for something to go wrong, and rolling it back is a much bigger job. And dropping many/bigger changes to the user experience is much more noticeable.

Again, this isn't the right process for everyone, but it works for us and its how we've managed to build a product that delivers value to our users.

[chrisandchris]

It's interesting how you differ between "deploy ASAP, within half a day" and "keep the fix a few months back".

Like, is there nothing in between? Like once a week, once every other week?

If you would have to estimate, wouldn't be there less bugs if you would deploy less fast (and use this time for validation)?

[sdairs]

Sure :D

I've worked on products across the spectrum, Enterprise software that releases twice a year, smaller stuff that ships once a month, and now a SaaS that releases many times a day. I don't have the data to compare, but I genuinely don't believe the rate of bugs was materially different between any of them.

We make faster changes, but the changes are much smaller, and so the surface area to test is smaller. The more changes you make, the more time you need to validate. If you make a weeks worth of changes, you're going to need an appropriately longer validation cycle than if you make 1 change. It scales, and in my experience, many products with slower release cycles aren't appropriately scaling up the validation time to match.

That said, this doesn't necessarily mean we're writing code and 30 minutes later its in production...there's still an iterative dev cycle with lots of validation happening...but if something is ready to go, ship it!

[Scubabear68]

Thanks for the response Al!

Very interesting. I agree different strokes for different folks, you guys seem to be on the extreme end of CI/CD.

Have you done any sort of analysis you could share on what it costs to release up to 20 times per day?

[sdairs]

We do track the real $ cost of time & materials, but tbh I don't think it's anything too exciting. I'll see what we can share!

[fragmede]

> multiple production releases per day

Most (but not all) SaaS businesses are expected to these days, so I'm curious what your business/industry is that not only that you don't, but that it gives you the shudders.

[Scubabear68]

I consult across a few different industries, but it includes SaaS offerings. Many places could do multiple deploys to prod a day, but choose weekly releases or other longer cadences. This is to allow for documentation, client notifications, etc. It also is more efficient, constantly releasing requires a lot of resources, as this blog implies they are churning dozens of K8 pods several times a day.

I would think constant releasing would also make debugging prod issues pure hell.