[eropple]

Terraform, as an idea, is brilliant. Mitchell and company isolated a hugely important need and tried to fill it, and I give them all the credit in the world for that. Cross-platform cloud provisioning? Gimme. But I cannot in good conscience not relate what a disastrous experience Terraform has been for me at both jobs and clients.

Writing reusable code in Terraform is an exercise in frustration due to the extreme clumsiness of HCL (which, I understand, was used because "YAML is complicated"--well, that's true, but YAML isn't a good solution either, you're HashiCorp, you wrote Vagrant, you already know how to do this!). The application architecture is reckless and full of race conditions; your state will be hosed if one resource errors out at the wrong time, while other resources are being successfully updated--the resources that return successfully after the failed resource will on many occasions fail to be persisted to state. What's more, application testing seems to be at best an afterthought: there have been regressions in the providers that will break your existing states.

I would under no circumstances use Terraform if I didn't have clients who had selected it before I was working with them. If in AWS, I would use CloudFormation, with a tool like Cfer[1] (which is excellent, reliable code) or SparkleFramework[2] (which is more full-featured but I hope you never need to debug it) to provision my stuff.

(Full disclosure: I'm building a much, much better provisioner for multi-provider cloud infrastructure. Neither of the projects I recommend are mine; mine's not done yet.)

[1] - https://github.com/seanedwards/cfer

[2] - http://www.sparkleformation.io/

[jacques_chester]

If you're writing your own, you might also look to BOSH[1] for inspiration.

It's older than CloudFormation and Terraform (born 2010). It can manage anything that someone's written a driver for. So far that includes AWS, Azure, vSphere/vCloud, OpenStack, VirtualBox, Google Compute Engine, Apache CloudStack and there might be others I missed.

It stores state in a database. It is able to recover from mismatches between the state of the world and the desired state. Cloud Foundry users have been using it for years to deploy and update CF installations. Pivotal Web Services (I work for Pivotal, in a different division) has been upgrading to the most recent CF release every few weeks, live, without much fuss, for years.

For any kind of heavily stateful infrastructure, BOSH is a strong candidate.

[1] https://bosh.io/

[eropple]

Augh, how did bosh slip my mind? I've never used it in production, but I've used it to roll out a CF environment for testing and was impressed to dig into it a little more (most of a year ago now, I think your mention of CF was actually what kicked that off). From (admittedly limited) experience I'm not crazy about its developer-facing feel, but I appreciate the significant and responsible effort in it.

[sagichmal]

> The application architecture is reckless and full of race conditions

Honestly curious, can you point to one or two?

[eropple]

I don't have a toy example offhand, but the resource failure case I mentioned is one. If resources A, B, and C are in flight at the same time and A fails, Terraform in some as-yet-undiagnosed circumstances will not record state changes caused by the still-in-progress work on B and C. This happens a lot with SNS queues, IIRC, because SNS queue operations on the AWS API take a relatively long time to resolve. So if, say, you mistyped an attribute for an EC2 instance, it can fail out and Terraform will happily forget that it created an SNS queue for you.

I have a sneaking hunch that the continuing problems with template-file resources (complaining that the "rendered" attribute doesn't exist in dependent resources) are related to this, but can't prove that; my clients don't pay me to debug Terraform, but to get their stuff working, and that doesn't leave much time to get in-depth with it now that I've decided not to use it for my own purposes anymore.

[duggan]

> Full disclosure: I'm building a much, much better provisioner for multi-provider cloud infrastructure. Neither of the projects I recommend are mine; mine's not done yet.

One of the convenient things about software that doesn't exist is that it doesn't have any bugs.

Let your software speak for itself when it exists; until then, this seems an undeserved critique of software, and a team, that is solving problems every day.

[tptacek]

This is a crazy sentiment. I was just considering checking out Terraform, and I'm really glad to have read the previous commenter's experience.

[Gigablah]

I've been using Terraform for over a year, maintaining a standard 3 AZ load balanced production cluster.

HCL has improved dramatically, and now that template strings are a thing, most of my variable interpolation issues are solved. However you still can't specify lists as input variables so you frequently have to resort to joining and splitting strings. It's hackish and worse, changing one value in the list will invalidate all other resources that use the variable.

Race conditions and dependency cycles are still a problem. Particularly with auto scaling groups and launch configurations -- I have to migrate them in two steps (create then destroy) to avoid a conflict. Same with EBS volumes, I ended up scripting my instance to attach the volume by itself, otherwise there's ordering issues when destroying and replacing.

There's also missing features, such as the ability to create elasticache redis clusters and cloudformation resources.

I'm still glad that I went with Terraform though. It takes a good amount of time to get around the limitations and bugs, which can be really frustrating, but when it works, it works beautifully.