The big takeaway for me here is that this “provisioning service” had enough internal dependencies that they couldn’t bring up new nodes. Seems like the worst thing possible during a big traffic spike.
> We run a service, aptly named ‘provision-service’, which does exactly what it says on the tin. It is responsible for configuring and testing new instances, and performing various infrastructural housekeeping tasks. Provision-service needs to talk to other internal Slack systems and to some AWS APIs.
The "configuring and testing new instances" part also sounds very fishy to me. Configuration should be done when creating the image and launch template, while testing should be the job of the load balancing layer. Why do we need a separate "provision-service" to piece everything together?
The "configuring and testing new instances" part also sounds very fishy to me. Configuration should be done when creating the image and launch template, while testing should be the job of the load balancing layer. Why do we need a separate "provision-service" to piece everything together?