| > We all run puppet / ansible multiple times a day on our infra right? Checking for config drift. No, not necessarily. I think a lot depends on the rate of change in your environment and the size of the team managing it, and the number, complexity and types of devel/test/production environments being managed. In an emergency services network like mine changes are rather the exception than the norm and running ansible/puppet in a loop every 30 mins is a waste of resources. Also, changes that break when pushed to production could result in fire engines or ambulances not being dispatched. Not good. A devops team I worked for ran up a full vagrant VM on every commit to the puppet repository, then ran puppet inside the VM and ran the full gamut of testinfra tests as well. The whole process took 20-30 minutes at times and if the new code you just pushed broke the tests right at the end it would be 20-30 minutes before you found out. Of course you were supposed to mitigate this by running the tests against your new code in a VM before you pushed them. So that is the other extreme. Personally I found that to be overkill, although it didn't stop broken changes being implemented to production sometimes. > Ansible returns changes to what they should be at the same time? Well yes so does puppet, but that assumes that your puppet plays/ansible playbooks etc. are all written to be idempotent. The default modules generally work in that manner, but both allow you to write plays/playbooks that aren't idempotent and that can break things. Infrastructure tests allow you to seperate out (or augment) the validation that running puppet/ansible gives you. Because the tests are generally of a read-only nature there is a less chance of accidentally changing the state of the servers when you run the tests. You could run ansible or puppet in 'dry-run' mode I suppose, and examine the output for errors, but testinfra or serverspec give you a much nicer interface IMHO and are more lightweight and execute much faster than a full ansible/puppet run. |