[cyberax]

First, you can guard against it. Just periodically re-run the infra code in the "dry-run" mode (from a CI/CD system) and scream if you see any differences.

Second, this is still fine. Don't make changes outside of the IAC control. And if you do make them, retro-fix the IAC files until there is no diff with the actual state.

Third, IAC should have an option to ignore some changes.

> Also, dynamically creating state creates all sorts of concurrency issues, which is another nice thing about stored state, you can put a lock on it.

In my experience, this is not a big issue in practice. Production deployments should be done through some kind of CI/CD, and it naturally serializes builds.

However, nothing stops you from adding locking without doing the full state management.

[substation13]

> Second, this is still fine. Don't make changes outside of the IAC control. And if you do make them, retro-fix the IAC files until there is no diff with the actual state.

This doesn't work in practice. Some aspects of the business want to tweak things and it should be reasonably guaranteed that the automated side never touches it.

Terraform state gives this assurance because it won't destroy resources not under its state.

[gregmac]

> Some aspects of the business want to tweak things and it should be reasonably guaranteed that the automated side never touches it.

What would a legitimate case for this be?

It seems to me like any changes either must be done via IAC -- and tracked in source control, PR'd, tested in non-prod, etc -- or a missing feature.

If there's a legitimate case for modifying something not in IAC, it should be supported -- this is what I mean by "missing feature". The app and/or IAC should have code for that feature.

Modifying IAC-deployed settings is akin to someone hacking the binary of an executable from a software vendor while still expecting the vendor to support that modified executable. Not gonna happen.