[pondidum]

> Do not make production changes on Fridays

I ~hate~ dislike this advice. If you can't deploy on a Friday, you need to fix your deployment strategy. By removing Friday from when you can deploy, you're wasting 1/5 of your available days.

Note: deploy != Release[1]. Use flags, canaries etc.

[1]: https://andydote.co.uk/2022/11/02/deploy-doesnt-mean-release...

Edit: hate is far too stronger word for this

[Sevii]

The point of not deploying on friday is to reduce the risk of getting paged over the weekend. It's a quality of life move for the oncall team. No deployment strategy will change the fact that deployments are the leading cause of outages.

If you can't afford to give up 1/5 of your available deployment days you have a problem somewhere in your CI/CD system.

[nijave]

Sure but ideally you have high enough confidence in your software that those types of issues are highly unlikely.

[maxfurman]

Over a long enough time, with enough deployments, unlikely things are guaranteed to happen! In the long run, setting a given feature loose on Monday instead of Friday is unlikely to move the needle in terms of business results, and eliminates a category of risk. I'm not superstitious about much, but No Friday Deployments is one of my rules

[rexarex]

I agree. We have some advocates that really want to say we deploy on Friday, but even they don’t seem super pushy about a Friday deployment when they have weekend plans, and the org as a whole isn’t keen either. It’s high risk and low reward. I’ve also burned myself on this more than once.

[kevan]

I'm a huge advocate for CI/CD pipelines and my team owns a lot of them. We're confident enough to deploy anytime but we choose to limit deploys to our team's business hours and not on Fridays. Why? Because we think the return going from deploying 4 days/week to 5 days/week is outweighed by the stress and morale hit of ruined weekend plans if something weird happens. There's probably situations where that extra speed makes a difference but for us deploying to all regions safely can take a full day anyways so it's pretty normal to have multiple changes flowing at the same time.

[pondidum]

I understand that, but would counter with that it sounds like deploy == release, and that if that weren't the case, you could deploy more often.

However, I will admit it is a trade-off; some engineering time does have to be spent to get there, and perhaps that engineering time is better used elsewhere right now.

[snowwrestler]

If you put new code into production, that is a deployment and an engineering release.

If some of that new code is a flag that controls what customers see, then from a product perspective it’s not a “product release”. Is that what you mean?

The point is, even feature-flagged code can break things unexpectedly.

[_vertigo]

What is the difference between deploying and releasing? Everywhere I’ve worked, those are the same thing. We deploy probably 10 to 30 times a workday. As for Friday deployments, I tend to avoid deploying big changes on Friday but medium and small are probably fine Friday unless it’s the very end of the day.

[pondidum]

From the link further up the replies:

> Deploy is the act of moving software into an environment and running it.

> Release is the process of making a feature visible to a user or subset of users.

In our case, we deploy often (10s of times per day), and make use of feature flags for releasing. Keeping an eye on data in Honeycomb before and after a deployment and before and after a flag is changed is part of the engineers job.

If something doesn't look like it's working as intended, ideally it is disabled with a flag, rather than needing to figure out how a rollback can be done (often it can't), or rushing a fix and roll forward.

[FridgeSeal]

This is a pretty small distinction in my experience, and I think you'd find that most people would go with "if it goes into prod, it counts as a release".

Regardless of how confident you are on your feature-flagging functionality, everyone's going to be unimpressed if suddenly your "minor" deploy has caused a bunch of things to go haywire at 4:30pm on a Friday and now people have to stay back and fix things.

[grogenaut]

CI/CD, flags, canaries don't catch everything, and can still cause outages to others. We try and do pretty heavy CI/CD where I work, but not everyone does (we, like everyone, has old systems). It's actually quite easy for us to have the well behaved systems honor release hours or not depending how their release history has gone, or coverage,etc... but they're well behaved, so they usually have great tests, and they're not usually panicked about rolling out after hours, they have their sh*t together.

The reason we have core hours release only without director approval (aka director approval required outside core hours) is so you don't piss off another team by paging them after hours, and so you aren't trying to shove out a thing on a system that doesn't have good coverage or by turning off the safeties. In a large company I've noticed many engineers assume urgency even where there isn't. As an approver myself, most of the time someone wants to rush is because they've not even had the convo with their manager on if it's worth the risk, they are assuming urgency because that's when the sprint ends or what some TPM added to a jira ticket 4 months ago.

I admit that sounds risky itself (the engineers not having the right risk training) but this is why we have a policy and tooling... most of the times I've dug in they're just very new and worried about perception as a new employee, so my job is to shepherd them through having that convo with their managers which inevitably has the managers saying "yes it can totally wait till monday", and the change is inevitibly a bit more hot than it should be due to accidental deadline pressure.

[rexarex]

I get that people really want to flex that they can deploy on Friday afternoon and NOTHING CAN GO WRONG, but it’s still foolish and flaunts Murphy’s Law. It can wait.

[nikau]

Plus they are likely running simple Mickey mouse systems that aren't intertwined with a bunch of other systems maintained by other groups.

[lockedinspace]

Yep, let's not forget that Murphy's Law, whenever you least expect it, boom.

[dopylitty]

This one made me laugh. I've been places that only allow deployments on Fridays because it gives the whole weekend to fix things if they break.

It's a good interview question as a candidate. If you ask the interviewer when they deploy and they say only Friday (or worse only once a month) then perhaps look elsewhere for your own sanity because it's a sign of serious malfunction either organizationally, technically, or both.

[spc476]

> or worse only once a month

Don't discount a job because of one deployment per month---it really depends upon the service. I joke that a busy year for me involved four deployments to production, but "production" for me wasn't a website, or even a web-based service, but a service involved in the call path for phone calls. Our customer is [1] the Oligarchic Cell Phone company and the SLAs are pretty severe.

I do have to ask---where do you people work where you have multiple deployments per week (or even per day)? To me, that sounds insane!

[1] Still is, even though I left a few months ago, and not because of the lack of deployments, but the shoving of Enterprise Agile [2] on the company by new management.

[2] Which is anything but Agile.

[fragmede]

Any SaaS company that that has trusted automated testing on the backend, big enough to have a platform team so that there's a continuous deployment (CD) system should be able to do that. It's less about the actual cadence and more about the fact that, given a bug or request for a new feature, developers are empowered to go and do a release.

[bityard]

I am on a tools team, so the "customers" for our team are the company's developers. For changes that might cause an extended outage if things go sideways, we generally prefer to do those after-hours or on weekends so that we don't have all the dev teams sitting idle during work hours if something goes wrong on our end.

The upshot is that this is fairly rare and we do not have an on-call rotation. If most anything breaks over the weekend, nobody is going to notice or care until Monday morning rolls around.

[fragmede]

Depending on your role, that is. If your desired position is straight dev with minimal to no ops work as possible, then yeah, red flag. However, if you're an SRE/DevOps-type person, setting up a continuous deployment system so they can deploy more often than that is a perfect landing task to dig your teeth into. Different strokes for different folks.

[tbrownaw]

If there's a very strong "only during standard work hours" usage pattern with SLA penalties for downtime, adapting deployment patterns to that reality can maybe be sensible.

[rexarex]

I love this idea.

[doctor_eval]

You should have both the confidence that you could deploy on Fridays, and the wisdom to know that you shouldn’t.

[lopatin]

Interestingly my company only deploys on Friday because it has to wait for (most) markets to close for the weekend.

[pondidum]

Oh now that is an interesting take!

Would that still be required if a deployment and a release are decoupled entirely, or is it unavoidable? Genuinely interested!

[lopatin]

Not totally sure of the distinction. We have versioned weekly releases and they roll out to prod on Friday.

[elric]

Hating it seems a little strong. I'm sure that any team far along enough on the quality spectrum can just read this and say "we've moved beyond this worry". The post is titled "general guidance", not "absolute truths". Adjust expectations accordingly.

[pondidum]

You're right, hate is far too stronger term for this, I've updated the wording. Thanks.

[charcircuit]

There isn't a bottleneck in the amount of commits you can ship per day. You just get more changes to roll out on Monday.

I do disagree with the absolute statement of not doing it, but I definitely do a risk analysis whenever I ship a change on Friday and avoid anything risky and just push it off until Monday morning.

Not all changes can be put behind flags and canaries don't really fix the issue (unless you are okay in blocking important fixes from being rolled out due to your bad change killing the canary)

[lockedinspace]

Seems reasonable, but if you work for a large company, you can't guarantee that a major release (which is a production change) won't cause any unexpected harm. I have worked with quite bit organizations, and deployed on Friday and wasted my entire Friday-night and Saturday morning, rolling back the +130 components that an app had.

If you are a small company, or you do not do extra weekend shifts, I understand your point. Elsewhere, you just want to live an adventure every Friday.