Hacker News new | ask | show | jobs
by jpdb 809 days ago
And if you're pairing your infra-as-code with a gitops model then you can help prevent these kinds of issues with PRs.

You can also use your git history to restore the infrastructure itself. You may lose some data, but it's also possible to have destroyed resources retain their data or backup before destroy.
2 comments
zaphar 809 days ago
The problem with infra-as-code and gitops is that it's often nearly impossible to tell what will actually happen with a PR without running it somewhere. Which is 1. expensive. and 2. nearly impossible to get to mirror production.

Production and staging are the farthest you can get from pure immutable environment that you can get. They carry state around all over the place. It's their entire reason for existing in some sense.

This means that while git-ops can be helpful in some ways it can also be incredibly dangerous in others. I'm not entirely sure it doesn't all come out in the wash in the end.

link
zzyzxd 809 days ago
GitOps is just like "DevOps" -- you don't really know what it means to a specific org until you talk to them, because people interpret it differently based on their own understanding (or if they have a horse in this race).

To me it always means describing the desired state of your infra in structured data, storing that in git, and run controller to reconcile it against the actual infra.

If your GitOps engine has to compile/run the "code" to uncover the desired state, that defeats the purpose of GitOps and is no better than running your hand crafted release bash script in a CI/CD pipeline.

It should have never been called infra-as-code, but infra-as-data.

See "The Rendered Manifests Pattern": https://akuity.io/blog/the-rendered-manifests-pattern/

link
zaphar 808 days ago
This does not change my statement at all though? You fundamentally can't really predict the impact of some changes in a given environment until it's deployed. Just because you can obtain the current state of the environment and reconcile some stuff doesn't change this.
link
zzyzxd 808 days ago
That's why you should call what you store in git the _desired_ state, not anything else. A git repository is not a live database. It's a collection of static text files that change less often than your live system. There will be bugs and misconfiguration, and sometimes the desired state is just technically not reachable, and that's fine. What the actual state is doesn't matter. Leave that to the controller. State drifting is a problem your gitops engine should detect, and should be fixed by the owner of controller code.

Some companies practice infra-as-code, point to their git repo and tell me "this is our single source of truth" of our infrastructure. And I have to tell them that statement is wrong.

link
zaphar 806 days ago
This is correct. You need some kind of running check on the environment and when possible code that handle exceptional cases.

Sometimes that's as simple as a service that shoots other services in the head to restart them. Othertimes it's more complicated. But lot's of places can't afford to get more complicated than "alert a human and have them look at it".

link
jansommer 809 days ago
Yes, and usually you can make rules that prevent databases from being deleted again from IaC.
link