[jakejake]

I can't seem to get into the game of fully automated deployments to production. It definitely interests me but a few things always hold me back.

The first issue is that I've probably set up 10 or 15 app development and deployment "systems" if you will. I've found that it's very beneficial to automate the simple stuff but it quickly reaches a point of diminishing returns. A super-custom system always works great for a while until some big change or library upgrade or refactor or whatever comes down the pipe. Then we spend a ton of time resetting up everything. We have to keep the build system components up to date so it doesn't turn into an ancient mystery box. Sometimes an upgrade breaks the whole thing and then we're on stack exchange all day debugging a parser library or some other thing that we don't care about. Basically spending hours and days and weeks on the build system so we can have that sweet one-click (or fully automated) deploy.

The other thing is that we release frequently but we tend to double check everything before it goes to production. Our staging server is auto-deployed except DB changes which we do manually. Right now it's about 2-3 clicks for us to deploy to production and it works fine. We still do DB changes manually though. It takes a minute or two to deploy. I feel like the process encourages that final check that everything is cool.

I guess I'm nervous to set up something that deploys to production simply by adding a tag to a slack message or the git commit message. Should I get over myself? If I change my thinking is it possible that deployment to prod could be a non-event?

[wcummings]

>I guess I'm nervous to set up something that deploys to production simply by adding a tag to a slack message or the git commit message. Should I get over myself?

Sometimes adding some friction to the deploy process can be good IMHO. Continual deployment isn't good for every product or every team.

[dekz]

It can still get tested and smoke tested before deploying directly to production. If you cant trust your automated tests and smoke tests then you're setting yourself up to fail.

[officialchicken]

Very true, but what's the ROI for full automation?

If a typical 2-3 click deploy generously takes an hour, and they do 40 per year... then it would take 1 year to break even presuming that a fully-automated system could be built and deployed in 1 man-week. If the deploy takes 10 mins, can it be built in less than 1 day?

Ignoring the development time for a fully automated system, I think the real question is, "how does a rollback and unscheduled downtime impact the ROI due to unforeseen problems?" because it will happen, eventually.

[brianwawok]

Well how many mistakes do they make with those 4 clicks? That's part of the point of automation - remove chances for humans to flub.

[reddit_clone]

Why downvote this? This is a valid point.

Automation is not just for saving time. It is also preventing the system from human errors.

People make errors due to fatigue, inattention etc. performing even simple tasks.

[makecheck]

I think it is OK to add tools to the process as long as the most critical parts are the least mysterious.

For instance, in a networked file system, the on/off switch for “move from version A to version B” ought to be about as simple as swapping a directory symbolic link. That way, everyone can see exactly what it is pointing to now, what it used to point to (as old version targets are probably in the same parent directory), and anyone can figure out how to roll it back instantly.

Given trivial on/off switches, the details of the rest of the system can start to gain complexity.

Also, it helps a lot to have something like a “beta flow” that is essentially a parallel replica of your production environment.