Hey Mitchell, congrats on the new announcements, great stuff! Out of curiosity, how are you building and operate HCP? Are you running it on top of Kubernetes or Nomad, or you're doing some other custom stuff?
- Full HashiCorp stack (Nomad, Consul, Vault, Terraform)
- Cadence (https://temporal.io/)
- Microservice architecture over gRPC and Consul Connect
- All services written in Go
- Customer clusters are created/managed by programmatically running Terraform using just-in-time cloud credentials from Vault
- All internal TLS certs for customer clusters dynamically created using Vault
- All external TLS certs for customer clusters dynamically created using LetsEncrypt via Terraform
- Frontend is Ember
> Customer clusters are created/managed by programmatically running Terraform
I have soooo many questions about best practices doing this. I run a service that needs to dynamically provision AWS resources, and lacking a clear path to do this programmatically, I shell out to Terraform.
* I assume you aren't shelling out :). Do you have any additional helper libraries on top of the Terraform code base to make it more of a a programmatically consumable API, as apposed to an end user application?
* Are you still pointing at a directory with resources defined in HCL, or are the resources defined programmatically?
* What are you using for state storage?
* What is the execution environment for the programmatic Terraform process? Since Terraform uses external processes for plugins, I've hit some issues with resource constraints around the max number of process sysctl's in containerized environment where I have multiple Terraform processes running in the same container.
Yeah this isn't very easy to get right at the moment so there is not going to be any silver bullet here. We had to iterate on our runner a lot to get this right, but we have a lot of experience since we do this for Terraform Cloud too.
Answering your questions:
> * I assume you aren't shelling out :). Do you have any additional helper libraries on top of the Terraform code base to make it more of a a programmatically consumable API, as apposed to an end user application?
> * Are you still pointing at a directory with resources defined in HCL, or are the resources defined programmatically?
HCL mixed with the JSON flavor of HCL for programmatically generated stuff. Variables in JSON format also programmatically generated.
> * What are you using for state storage?
We output it to a file and handle this in an HCP microservice. We encrypt it using the customer-specific key with Vault and store it in a bucket that only the customer-specific credential has access to. If there is an RCE exploit somehow in our workflows, they can only access that customer's metadata.
> * What is the execution environment for the programmatic Terraform process? Since Terraform uses external processes for plugins, I've hit some issues with resource constraints around the max number of process sysctl's in containerized environment where I have multiple Terraform processes running in the same container.
Containers in HCP and VMs in Terraform Cloud due to increased isolation requirements. HCP has less strict requirements because the Terraform configs and inputs are more tightly controlled.