[nineteen999]

Belts and braces approach. I use Ansible but I also use testinfra (which is similar to serverspec and I suppose goss as well) to validate things that are not explicitly covered by Ansible, and even in some cases, some things that are.

The tests can be run independently at any later date, helping to ensure that admins haven't messed with important files, upgraded random RPM's or that the servers haven't suffered any other type of configuration drift.

But then I build/run emergency services infrastructure with 99.999% availability targets.

Once you are writing tests alongside your Ansible playbooks and committing them to source control, it doesn't really take a lot longer and eventually you have a test suite you can run across your entire environment at a moments notice.

[devn0ll]

"The tests can be run independently at any later date, helping to ensure that admins haven't messed with important files"

We all run puppet / ansible multiple times a day on our infra right? Checking for config drift.

I'm used to a banking sector where we have extremely stringent demands. And still I see no added value for something like infratest/goss.

Seriously: running ansible on your infra checking for config drift.. is that not exactly the same as running goss? Plus: Ansible returns changes to what they should be at the same time?

[nineteen999]

> We all run puppet / ansible multiple times a day on our infra right? Checking for config drift.

No, not necessarily. I think a lot depends on the rate of change in your environment and the size of the team managing it, and the number, complexity and types of devel/test/production environments being managed.

In an emergency services network like mine changes are rather the exception than the norm and running ansible/puppet in a loop every 30 mins is a waste of resources. Also, changes that break when pushed to production could result in fire engines or ambulances not being dispatched. Not good.

A devops team I worked for ran up a full vagrant VM on every commit to the puppet repository, then ran puppet inside the VM and ran the full gamut of testinfra tests as well. The whole process took 20-30 minutes at times and if the new code you just pushed broke the tests right at the end it would be 20-30 minutes before you found out. Of course you were supposed to mitigate this by running the tests against your new code in a VM before you pushed them. So that is the other extreme. Personally I found that to be overkill, although it didn't stop broken changes being implemented to production sometimes.

> Ansible returns changes to what they should be at the same time?

Well yes so does puppet, but that assumes that your puppet plays/ansible playbooks etc. are all written to be idempotent. The default modules generally work in that manner, but both allow you to write plays/playbooks that aren't idempotent and that can break things.

Infrastructure tests allow you to seperate out (or augment) the validation that running puppet/ansible gives you. Because the tests are generally of a read-only nature there is a less chance of accidentally changing the state of the servers when you run the tests.

You could run ansible or puppet in 'dry-run' mode I suppose, and examine the output for errors, but testinfra or serverspec give you a much nicer interface IMHO and are more lightweight and execute much faster than a full ansible/puppet run.