[DougBarth]

We've been using Puppet to manage our servers for some time now. As a group of developers doing our own operations work, we've found puppet both good and bad.

Setting up puppet was relatively straight forward. We had the puppetd auto-updating our servers for a while, but ultimately decided to manually run it when deploying changes. Managing zero-downtime changes was more error prone with it running.

Some aspects of Puppet have over time proved frustrating to us. The top annoyance is we never quite figured out a good way to test our puppet changes before checking them into git to deploy them to our puppetmaster. That has lead to a number of "fixing errors" type commits. The second annoyance we've found is actually highlighted as a feature: no implicit ordering of operations. While it might sound great to be able to reorganize your configs without fear of breaking the deployment, we've found that the tradeoff is that you don't find out that your configuration doesn't define its dependencies correctly until you try to kick a new server after spending months incrementally adding to your existing servers. For us, at least, having an implicit top to bottom ordering would lessen that headache.

Despite some of these headaches, simply having our configuration in version control is a huge win for us. We can setup a box much more easily, and we have a comment trail of why changes were made.

[Goladus]

About testing puppet changes...

If I had to do it again I would probably ditch the puppetmaster altogether and use an rsync server to distribute the entire configure repository to every server, then run puppet locally to apply changes. This way you can simply modify any local repository and run puppet to apply the desired configuration to any machine you want. When you're happy with the changes you can check them in.

Using the puppetmaster and the puppet fileserver was trickier, essentially I would use FACTER_var="value" to pass in a value to puppet that would use local files rather than central files (which came pretty close to the purely decentralized model anyway).

[wfarr]

You may be interested in checking out Rump: https://github.com/railsmachine/rump

[Disclaimer: I work for Rails Machine.]

It's essentially a headless puppet that centers around a workflow of testing changes from an individual checkout of the puppet code on a target server, testing no-op applies of the manifests, and applying the manifest until you're happy enough to commit, push, and roll-out.

This won't help with your second annoyance, sadly, but it should definitely help with the first in quickly pinpointing these sorts of issues without having a messy commit history.

[asenchi2]

Regarding your troubles with "ordering of operations". I've found it varies on installation, but that each team works to avoid these issues by setting a standard on module development. So that when you "include 'ntp'" you know exactly what you are getting. I've seen many different ideas on how to accomplish this, all of which made it really easy to include without ramifications.

Also, regarding testing. I think this is an issue with both Chef and Puppet. Something I hope someone addresses at some point in the future. I've seen some custom tools with some promise (Chef focused) but perhaps a Vagrant setup might be the best answer these days.

[bostonvaulter2]

Regarding the non-ordering of dependencies I wonder if it would help to have some sort of shuffle flag. I know that redo has implemented a --shuffle flag to tease out missing dependencies.

https://github.com/apenwarr/redo/blob/master/Documentation/r...