[xorcist]

That's the whole idea of using a system based on declarative state. As soon as the system is back up, the agent can resolve state again. You also keep a central copy of the state of every agent.

You can absolutely do this by writing your own agent (or by writing a family of bash script, but they tend to grow pretty complex over time), ansible is just a framework to write that in a standardized way. It will also out of the box handle a number of common system state such as running services and sysctl triggers.

There are a number of similar systems such as puppet or salt, which are all variations of the same basic idea. 30k hosts are a lot, and will need sizing the system appropriately, but it's not an unusual configuration by any means.

[latchkey]

That was the benefit of my system over what you are talking about, there was no dependency on a global state or centralized control surface. Each worker was autonomous and self contained and had enough intelligence to bring itself to the desired state on its own. All you had to do was one line curl|bash install my service and it would take care of the rest without any other external dependencies. No worries about having to have ansible try to connect over and over again until things were working. It would just magically fix itself.

Again, many ways to skin the cat, but at the end of the day, this solution really worked extremely well. I would do it again in a heartbeat.