[gossamer]

I kinda like the way ansible does it. There is a concept of a vault. You put all the passwords in that file and they are all encrypted. You use one password when running the command or playbook and all of the keys are decrypted as needed.

I don't know if that is efficient for 30K machines though.

[latchkey]

It isnt. I ended up building a small golang binary that ran as root and I could hit it with http calls to execute whatever I wanted. Built a message queue that would work through all the machines for eventual consistency. Worked great.

[xorcist]

I've used both puppet and ansible at that scale and it's not only workable, it is a good idea.

[latchkey]

How did you deal with machines randomly crashing and ensuring everything was consistent at all times across multiple physical datacenters?

I was also solving more problems than just ansible/puppet solve... like monitoring of specific systems on the machines.

Regardless, more than one way to skin a cat.

[xorcist]

That's the whole idea of using a system based on declarative state. As soon as the system is back up, the agent can resolve state again. You also keep a central copy of the state of every agent.

You can absolutely do this by writing your own agent (or by writing a family of bash script, but they tend to grow pretty complex over time), ansible is just a framework to write that in a standardized way. It will also out of the box handle a number of common system state such as running services and sysctl triggers.

There are a number of similar systems such as puppet or salt, which are all variations of the same basic idea. 30k hosts are a lot, and will need sizing the system appropriately, but it's not an unusual configuration by any means.

[latchkey]

That was the benefit of my system over what you are talking about, there was no dependency on a global state or centralized control surface. Each worker was autonomous and self contained and had enough intelligence to bring itself to the desired state on its own. All you had to do was one line curl|bash install my service and it would take care of the rest without any other external dependencies. No worries about having to have ansible try to connect over and over again until things were working. It would just magically fix itself.

Again, many ways to skin the cat, but at the end of the day, this solution really worked extremely well. I would do it again in a heartbeat.