[karmarepellent]

> while monitoring configuration compliance with a custom Naemon plugin.

While I absolutely agree with you and your approach, would you mind elaborating what kind of configuration compliance you are referring to in this statement? I suppose you do not mean any kind of configuration that your Puppet code produces as that configuration is "monitored", or rather managed, by Puppet.

[ghomem]

I don't mind elaborating - the fact that people are asking me questions reminds me that I need to invest a bit more effort on some articles.

This case is actually pretty simple.

Puppet applies the configuration you declare impotently when you run the Puppet agent: whatever is not configured gets configured, whatever is already configured remains the same.

If there is an error the return code of the Puppet agent is different from that of the situations above.

Knowing this you can choose triggering the Puppet agent runs remotely from a monitoring system, (instead of periodical local runs), collecting the exit code and monitoring the status of that exit code inside the monitoring system.

Therefore, instead of having an agent that runs silently leaving you logs to parse, you have a green light / red light system in regards to the compliance of a machine with its manifesto. If somebody broke the machine leaving it in an unconfigurable state or if someone broke its manifesto during configuration maintenance you will soon get a red light and the corresponding notifications.

This is active configuration management rather than what people usually call provisioning.

Of course you need an SSH connection for this execution and with that you need hardened SSH config, whitelisting, dedicated unpriviledged user for monitoring, exceptional finegrained sudo cases, etc. Not rocket science.

[karmarepellent]

Thank you for your thorough explanation. Interesting to see that you basically use your monitoring system as a scheduler to run Puppet and it sounds beneficial to closely integrate it with your monitoring to have it all in one place.

At my place of work we went the "traditional" way of running Puppet locally. It has been our experience that Puppet failures due to user misconfiguration or some such do not require our immediate attention (e.g. after hours), so we just check Puppetboard a few times per day to identify failing nodes.

Another reason why we use Puppetboard to monitor Puppet nodes is that every alert that our Icinga monitoring system produces is automatically interpreted as an incident which needs immediate attention. We are currently in the process of changing that so we are able to process non-critical alerts in a saner way.

Anyway, interesting to see how a fellow Puppet user manages their setup. Keep it up!

[ghomem]

Thank you as well, for sharing these notes about your setup. Indeed concentrating everything in the same monitoring system is very helpful as it reduces the cognitive load. You can likely do the same with Icinga.

Feel free to reach out on Linkedin if you need some more details. More than happy to share.