I really, really want to like Terraform. I tried it on a medium project by starting with and modifying the Segment stack. I ran into constant problems with it not being able to deal with errors and not being able to modify the stack in a way that would work. I ran into several situations where I would have to change names and labels just so it could work around itself while I was trying to iterate on the stack and bring it to a workable state.
To give you an idea of how many problems I had I currently have 4 different tfstate files from 4 days of testing. I had to go into AWS and manually delete resources because it couldn't recover from the errors it created.
One example: I was using the ECS option and changed the container source for a service. Seems easy enough and something that should work. Terraform wedged itself after applying the change so badly that I had to blow away the entire setup to get it reset to where it could even run `plan` without erroring.
Otto looked nice but it had fundamental, basic issues and it seems like nobody was actually working on it. I +1'd a bug with the PHP implementation where it didn't give you the option to change the web root and never got an update. This is something that every single decent PHP framework out there REQUIRES and wasn't supported. Otto PHP seemed like it was designed simply to work with Wordpress.
Have you considered starting your own stack from the ground up?
I definitely find it a lot easier to manage and reason about if I mostly avoid third-party Terraform modules. Out of probably a dozen different Terraform projects, I've never run into a situation which I needed to manually resolve. This includes both projects which I started myself and cases where I'm helping to improve/manage client deployments.
> I was using the ECS option and changed the container source for a service.
What do you mean by this? I've used ECS with Terraform extensively and never had a problem with updating the container/image which a service referenced.
That being said, I never used Otto. It definitely seems like they tried to bite off more than they could chew and I wasn't really interested in such high-level solutions.
> Have you considered starting your own stack from the ground up?
Yes. I'm actually in the middle of trying that now.
I want to set up a vpc, a few web servers (1-10) with an autoscaling policy behind the vpc along with a bastion server and a cron server, a code deploy setup to work with autoscaling, cloudwatch logging and monitoring, a load balancer, an elasticache instance, and an rds instance. I've been working on this off an on for months. If you or anyone else can point me in a direction to simplify this I'd be grateful.
The core of the problem I had with terraform (outside of the ECS issue) is that there is one AWS service that gets soft deleted. I can't remember what it is right now but it really threw tf for a loop. So I'd setup the stack, do some testing, decide to shut everything down for the day with a `terraform destroy` and the next day i couldn't resume because tf thinks the resource exists but aws doesn't think it does.
What you are describing is pretty trivial with Terraform so shouldn't definitely take months. A week or two.
You can look at my github.com/RichardKnop/coreos-cluster as an example (that one sets up a CoreOS cluster but you can take just the VPC, RDS, security groups, subnets and NAT bastion from there. I also have couple more terraform repos on my GitHub that deploy AWS infrastructures like you described.
Also look at the GitHub of Government Digital Service (GDS), I think it's alphagov. They have a lot of nice terraform stuff there from their experimentation with different PaaS.
What part are you having trouble with? You can use the AWS provider for the VPC, security groups, EC2 instances for the bastion host, cron server and load balancer (or an ELB). Elasticache and RDS also require parameter groups and subnet groups. They don't support replication groups for Elasticache yet, but there's a PR for it.
I've also seen tfstate get weird after a slow Elasticache spin up or termination. If it takes over 10 minutes it times out. The main thing I don't like about Terraform is that they don't support conditionals, which can be annoying.
I'm going to throw one thing in here about terraform over boto / etc, it's actually really nice to have a way to put together your own modules. I think they could do a better job with some common libs to make it easier, but overall it's super powerful.
To give you an idea of how many problems I had I currently have 4 different tfstate files from 4 days of testing. I had to go into AWS and manually delete resources because it couldn't recover from the errors it created.
One example: I was using the ECS option and changed the container source for a service. Seems easy enough and something that should work. Terraform wedged itself after applying the change so badly that I had to blow away the entire setup to get it reset to where it could even run `plan` without erroring.
Otto looked nice but it had fundamental, basic issues and it seems like nobody was actually working on it. I +1'd a bug with the PHP implementation where it didn't give you the option to change the web root and never got an update. This is something that every single decent PHP framework out there REQUIRES and wasn't supported. Otto PHP seemed like it was designed simply to work with Wordpress.