[NikolaeVarius]

I find people dont know how amazing a "immutable" server fleet is until you've experienced it.

It was so trivial to terminate and restart dozens of servers at any given time since unless there was a mistake in the cloud-init, we could bootstrap our entire infrastructure from scratch within an hour.

It was amazing, never had to deal with something missing on a server or a config being wrong in a special case. Dozens of hosts just purring along with 0 downtime since the moment anything became unhealthy, hosts would start auto-booting and terminate the old instance.

[MonaroVXR]

"It was so trivial to terminate and restart dozens of servers at any given time since unless there was a mistake in the cloud-init, we could bootstrap our entire infrastructure from scratch within an hour."

Got any tips in this regard?

Ansible, Puppet, Python, Terraform, Openstack Pulumi springs to my mind. Still need to cover a PXE server. (Maybe with Ansible)

[NikolaeVarius]

We use Salt Masterless with Packer to build AMIs.Terraform defines our infra. No matter what happens we ensure any service can boot by itself with 0 intervention