[dmlittle]

At Lob we just finished migrating all of our AWS resources to Terraform. We have a strict policy where everything (creating, updating or deleting) should be done through Terraform and the AWS console should be used as a read-only dashboard (some actions on the console such as invoking Lambdas, deleting an SQS messages, etc. are fine). We have alerting setup for any action that is performed in our AWS accounts that was done through the console.

Right now, changes are applied manually by the terraform changes author (every engineer has access to provision infrastructure they need, some teams have access to more AWS accounts than others) but we're looking to move to an automated environment such as Atlantis or Terraform Enterprise later this year.

[vogtb]

Nice, that sounds ideal! I've always found it frustrating to use the AWS console for anything other than dashboards and one-off trouble shooting.

You don't ever have issues with teams not applying resource changes? I.e. contributor A adds a resource, but doesn't apply it, and then contributor B goes to apply their changes and sees unexpected resource changes that they potentially don't have the permission to create? I guess that would be as much of a cross-team problem as a technical one.

[dmlittle]

Most of the Terraform changes are done by the infrastructure team which is currently 5 people. We're pretty diligent about making sure that whatever is in our master branch in the repo is what the source of truth is and what our environment should look like. If there's anything that looks off it gets caught by running a plan before applying and we quickly sync up and make sure to push the changes upstream or override them. For the smaller changes that are performed by other teams, they need a stamp of approval from someone in our infrastructure team and we make sure to be proactive about being available to not block them as well as making sure the changes are applied properly.

That being said, our current approach will definitely not scale well and that is why we're going be looking into a CI/CD pipeline way of applying changes.

It's also worth noting that it took _a lot_ of effort to migrate everything into terraform. The infrastructure team took the initiative to migrate everything in a zero-downtime way such that we wouldn't be affecting the application teams.