[threeseed]

> Even if you have a great deployment pipeline, at some point, you'll need to add a missing index in production fast because a wave of users came in and revealed a shit query.

This sounds more like a CI/CD and process issue.

There is no reason why adding a new index in code and deploying it into Production should be more complex or error prone than modifying it on the database itself.

[mjr00]

Direct execution of `CREATE INDEX...` on a database table is always going to be faster than going through a normal deployment pipeline. Even if we assume your pipeline is really fast, which is probably not the case at most orgs, you are still comparing a single SQL statement execution, to a single SQL statement execution + git push + code reviews + merge + running through Jenkins/Circle/whatever. How long does that overhead take? How much money have you lost because your website won't load when your post is on the frontpage of HN? Seconds and minutes count. I don't want my code crashing because an unexpected index exists in this scenario.

[threeseed]

You should be able to deploy end to end to Production in less than a minute.

Companies should be focused on solving that problem first before doing insanely short-sighted workarounds like skipping pushing to Git and code reviews.

[mjr00]

> You should be able to deploy end to end to Production in less than a minute.

When I was at AWS (RDS) our end-to-end production deployment process was 7 days. We were also pulling $25million/day or so in profit. I'm sure that number is much higher now.

There's a large difference between what the theoretical "right" thing is from a textbook perspective, and what successful engineering teams do in reality.

edit: besides, it doesn't even make sense in this context. I have 100 servers talking to the database. I need to create an index, ok, add it to the code. Deploy to server 1. Server 1 adds the index as part of the migration process, and let's say it's instant-ish (not realistic but whatever). Do the other 99 servers now panic because there's an unexpected index on the table?

[jeltz]

I don't think I have ever seen a non-toy project where that was the case.

[kelnos]

That's a lovely ideal, but I'm the real world, there are relatively few companies that meet that metric.

[threeseed]

I've worked at FAANG and enterprise companies and we managed to do it.

There are no technical reasons why it can't be done. Only process and will.

[kelnos]

Yes, and that's exactly the point. The reality doesn't usually match the ideals, and many orgs do not have good process, and do not have the political will to get good process implemented. Part of being a professional is recognizing where reality falls short of the ideals (an all-too-common occurrence), and doing the best you can to successfully get your work done in that environment.

And of course I don't know which FAANGs you worked at, but I know folks at FAANGs who have complained to me about CI and deployment times. Hell, these are huge companies; while they try to harmonize tooling, deployment times (especially when test suites of varying quality are involved) can vary a lot across a company. I wouldn't be surprised if there were people at the companies you worked at that were upset with deployment times, even if the teams you worked on were in good shape.

Honestly, when someone suggests something like you've suggested (that everyone should be able to get their deployment times to under a minute), I really do wonder if they're intentionally arguing in bad faith or are trolling. I know for a fact that things are not that rosy, and are rarely that rosy, even at the companies you claim to have worked at, and it's hard to believe that anyone could genuinely think that this is a broadly-attainable target. That doesn't mean that no one can do it, but that does mean that designing tooling that assumes everyone can do it is... well, just kinda naive and not very useful.

[threeseed]

You have two choices: (1) try and solve your deployment issues or (2) make unmanaged, untested, unreviewed changes directly in Production.

Now you may say I'm just trolling but option (1) seems better to me for the long-term health of the project/company. And I don't believe it's correct to say it is an unrealistic goal.

[swiftcoder]

> I've worked at FAANG and enterprise companies and we managed to do it.

You have a very different experience to the rest of us, in that cases.

The big AWS services all had deployments measured in days - or even weeks (depending on how many regions they are deployed across). Facebook's monorepo took upwards of an hour just to get a PR through the merge queue. Both were notorious for "hand-jamming" critical fixes directly to production.

[pclmulqdq]

There are lots of reasons to do slow rollouts. You should be rolling out in stages anyway.

[tempodox]

You do code review in less than a minute?

[threeseed]

You do code reviews/automated testing in lower environments before you make changes directly to Production.

And in this case if it's an emergency hot fix then it's still better to do this through a managed, tracked, tested pipeline.