[wgerard]

Not a slack employee but worked at a company with similar CD views:

(Likely) various groups of people are deploying to production throughout the day. Out of those 100 deploys, an individual is probably only involved in 1 or 2 a day. As soon as you're ready to deploy your code, you queue up and see it all the way through to production along with probably a few other people doing the same thing.

The actual "change the servers over to the new production code" process is usually instantaneous or extremely quick, the 10 minutes is mostly spent testing/building/etc.

People (including myself) enjoy this because you can push very small incremental changes to production, which significantly reduces the chance of confounding errors or major issues.

Note that this is would be a Sisyphean task if your company doesn't have great logging/metrics reporting/testing/etc.

[perpetualcrayon]

I once worked on a project for one of the largest corporations in the world. First part of my career I only ever worked on smaller teams.

I was excited for the move to a large corporation where there would be amazing room for growth and learning.

I have to say that almost a year into my work on this project, I was absolutely stunned how inept this company was at coordinating a technology project.

Something a small team could accomplish in a matter of months was taking 100's of developers and 100's more in supporting / operational roles years to accomplish. My guess is the developers on this project would gladly trade places with Sisyphus.

[toomuchtodo]

It’s usually a Sisyphean task. Everyone wants to look and act cutting edge (“but Netflix!”) but nobody wants to make the necessary investments in the tooling, org structure, and management ability/support that is required to support that sort of deployment cadence (if your org focuses on who broke something instead of the process, and management doesn’t want to change that culture, all hope is already lost [based on experience in a large enterprise, YMMV]).

There are some legitimate needs for continuous deployment, the rest of it is cargo culting.

[wpietri]

I think some organizations need it more than others. And I'm sure there's a lot of chasing the cool new thing. But I'd argue that almost every organization can benefit from continuous deployment and the discipline it requires.

The first time I switched from CI to full CD was circa 2011. I loved it because the mental bucket "later" went away. Except to the extent something was declared as a formal experiment in our A/B tests, code was either live or it wasn't. We were doing pair programming and committing every few hours, so aside from the little scratch-paper to-do list before we committed, there was no "later" for us. It made it real clear what our "good enough to ship" standards were. There was less room for bullshit. The resulting code was tighter, cleaner.

It also forced us to work much more closely as a team. We couldn't leave product questions for some end-of-iteration review. We had more mini-reviews with the product manager, and also improved our product judgment. Everybody trusted each other more. Partly because we had to, and partly because close collaboration is how you build trust.

It also shifted incentives further upstream. Suddenly there were no more big releases. No matter how big your vision, you had to learn to think of it in bite-sized pieces. It became less about answers, and more about questions. Not "Users need X!" but "What's the smallest thing we can do to see if users benefit from X?" Being able to make a lot of small bets made it easier to explore.

The Lean and Kanban folks talk a lot about "minimum WIP", where WIP is work in process. My CD experiences have definitely convinced me that they're right. Smaller pieces deployed more frequently requires a fair bit of discipline, but there are such huge gains in effectiveness and organizational sanity that I'll always try to work that way if I can.

[andrewaylett]

There's continuous, and then there's continuous. Where I work (which is easy enough to find out if you're curious, and the same order of magnitude size of engineering team as Slack) we deploy hundreds of times a day. But that's because our units of deployment are small (and target-specific, so a single logical change can trigger multiple logical deployments). So we're not deploying the same thing hundreds of times per day, we're deploying hundreds of things once a day.

Make a change to a thing, which might take a few minutes or a few hours, get it reviewed and merged and it'll be in production a few minutes later.

Once you get past 500 engineers, even just completing a single task a week means 100 things to deploy every day: either you batch them together somehow or you work on the tooling to just get them to production without any fuss.

[wgerard]

> There are some legitimate needs for continuous deployment, the rest of it is cargo culting.

Maybe, but I wouldn't go that far. Small companies already often do CD, because there's rarely a rigid deploy schedule. It's a practice people understand and feel the benefits of immediately. If you ask someone who moved from a small startup to a huge company what their biggest complaints are, I bet "longer/stricter deploy process" comes up 8/10 times.

When I think of cargo cult programming I think more of TDD or Agile: Practices that people aren't familiar with and often implement without understanding the benefits or reasoning.

[toomuchtodo]

For every developer that complains about the longer/stricter deploy process, I'd offer up for consideration deployments that went out through the CD pipeline where production data was mangled with no rollback possible. As with everything, its determining your appetite for risk.

[wgerard]

Hmm, I don't see how that changes with longer/stricter deploy processes - unless you have some of the tooling around that makes CD very possible in the first place (automated checks/etc.)

I've certainly worked in places with very long and strict deploy processes that managed to mangle production data frequently. Even worse, because the deploy process was so strict and long the bad code managed to stay on production for much longer than 10 minutes (the deploy time mentioned in the article).

There's some vague notion out there that long deploy process == safe, but there's very little evidence to suggest that's the case. If anything, it seems much more dangerous because larger changesets are going out all at once.

[toomuchtodo]

It goes back to my original comment above; if you have the proper tooling (tests executed that must pass prior to deploy given a green light, green/blue deploys, canaries, automated datastore snapshots/point in time recovery, granular control of the deployment process), I think continuous deployment provides a great deal of value above what you've invested into the process. But that investment is critical if you've bought into CD. Otherwise, it's "deploy and pray".

[wgerard]

Sure, and I guess my point is: If you haven't invested in those things, waterfall-esque deploy processes are just as bad and perhaps even worse because there's more chance for confounding changes to cause a nasty error.

The only reason waterfall-esque deploy processes work without those things is because companies often waste tons of people-hours on testing things out in the staging environment (which requires time, obviously).