Very excited to see this (if it is what I think it is...). Still reading about it.
I really like Terraform but haven't been a fan of the existing way of bringing external code into my terraform scripts, via some strange go-like "refer to this github" repo mechanism.
Hopefully this makes them package able artifacts. Looking forward to watching the keynote and other videos from Hashiconf.
I'd really love to be able to build modules in some other language like python, ruby or even javascript just because I am not very fluent yet in Go and its tooling.
If you want to build custom Terraform providers _without_ using Go, this is actually on the cards now: as of `hashicorp/go-plugin#fa4a2a351`, plugins can use gRPC to communicate with the core of Terraform. I don't know if anyone has implemented the primitives for building providers in another language, but it's now possible without too much pain.
Interesting, they are adopting the same pattern that I had discovered for AWS modules:
* all modules take in a var.name and var.tags variables.
* if the resource has a unique identifier, use var.name as a prefix (eg: "${var.name}-elb")
* all resources are tagged with the Name merged into var.tags
This allows to (1) avoid name clashes with the name prefix and (2) make sure everything is tagged and easier to filter by on AWS, while deferring most of the tagging decision to the user.
hashicorp-people: do you have any plans to open-source sentinel?
It seems like something that could have broader use, outside the hashicorp-suite.
For example, many SaaS-applications (say, a system used by some bank) usually end up writing their home-grown DSL for auth or use a language library. IAM also has something similar, with JSON payloads (but that's not code).
Would be interesting to know more about how sentinel is implemented under the hood.
This is absolutely useful. I keep a bunch of half-assed modules in a repo that I find common across projects. Love the idea of being able to find modules to kickstart common infrastructure.
Also Terraform is a great tool and the maintainers and contributors have done a fantastic job putting in useful features in the last few versions.
Imagine you want to create some new machines. Maybe you need four servers: 2 running Ubuntu operating systems and 2 running Centos.
Traditinally, you would get four physical servers together and then manually load an operating system onto each one.
With the advent of cloud technology, you can simply go to Amazon or DigitalOcean or whoever and just "create" these resources by paying for space in their data centers. You get four "virtual servers" instead of physical boxes you have to configure yourself.
Terraform essentially lets you "code" and automate the cloud provisioning. Instead of going to a dashboard and ordering four servers, you run a script that describes all of the servers you need as well as their operating system / networking / etc. needs. This scales very well (once you have a config you can replicate it to whatever quantity you require) and also provides consistency and reliability, as well as documentation, of your infrastructure.
Terraform provisions servers and infrastructure based on yaml config files that you can do things like commit to source control and see a diff of changes to your cloud provider. It's really fun but very tied into each specific service providers' nomenclature and services, I wish it was higher level. It provides backends for all the major players but I think there is (was?) some large variation in quality.
I find this pretty exciting. I imagine the most compelling use case for most orgs will be hosting their own private registry using Terraform Enterprise.
We already see with other "use this as a building block" public registries like Docker's where people blithely pull this stuff in and don't pay attention to whether or not it actually works according to even some semblance of best practices. So what's the handler for the "hey, this is broken/insecure, this needs to be fixed or taken down" case? How does this alert users whose infrastructure pulls this in?
HashiCorp, as some of the leaders of the Make Decisions On Five Minute Demos #squad, usually doesn't answer even half the questions for their Bright Ideas before people adopt them and find all the rakes on the lawn. This feels even worse than usual.
Building strong foundations matters. This doesn't help somebody who knows what they're doing do it better/faster--because of the inevitable impedance mismatches, in part because folks do this stuff differently and in part because Terraform is not particularly expressive and workarounds will involve destructive manipulation of modules--and it harms people who don't.
The product just launched but there is an early active testing component to the registry. I think they called them "verified" modules[1].
For the modules that we are publishing for Kubernetes[2] we are actively testing them and appreciated that the publishing process and bug reporting process are tied to GitHub. Between that and the verified modules I think it does add some signals of quality.
Now, don't get me wrong I think the DSL could be more expressive to make modules easier to configure and deploy. I am hopeful the project will get there.
I expressly am not worried about you guys or the modules you write--CoreOS is not my thing and I think it's barking up the wrong tree in a lot of ways but I appreciate that you guys actually care about a certain level of competence and security. But you know as well as I do that the first question when it comes to infrastructure is "okay, how is this going to break and hose me?", yeah? For something so critical to users' everythings, hard answers to process problems need to be front-and-center.
(Aside: the DSL will never get there while it's pretending to be JSON. At some point, they will bolt an actual scripting language on top and pretend that HCL was a bad idea to start with, or they'll be replaced with something that will.)
Hey Ed. Thanks for the feedback. You bring up some good points and I do want to address some of them. (And a note for some readers so they can understand my biases while reading: I'm representing Hashicorp officially)
In designing the registry, its API, its required module structure, etc. we worked for the past year with ~10 or so large community users, consultancies, partners, and customers. We had a representative from each that was actively involved in the design from the beginning, and they had a really big impact on what the registry ended up being. These are people who've written hundreds of modules, who know the best practices of their clouds, etc. So we hope with this collective knowledge we've built something that was longer than a five minute demo. :) They also had great feedback on various improvements we can/should make to Terraform itself and we are doing a lot of that with TF 0.11.
On the "works or not" topic: we provide verified modules for this reason. Verified modules are not only tested by HashiCorp engineers but also involve an active relationship with the module maintainer to ensure that the modules stay functioning across Terraform updates and cloud updates. These are recognizable with the blue checkmark. We launched with about three dozen of these but I expect community modules to lap this quickly.
At the same time, we don't want to actively disallow or heavily curate what people do with modules. So community members can publish pretty much anything they want on the registry. If you're using a community module, you're definitely taking your own risk and we hope we spell that out in the docs.
We already have plans underway to provide "module scorecards" in a way to help the community further recognize the quality of a module versus simply the blue check mark or no blue check mark binary approach. This design was part of our initial design, but isn't ready for launch.
Going beyond the public registry, we also announced today we'll have a private registry available in Terraform Enterprise. Note that the actual public API of the registry is documented (https://www.terraform.io/docs/registry/api.html) so separate implementations are possible and welcome.
But within our commercial private registry, we will actually track _who_ is using your modules. When you change the public interface to a module (inputs, outputs), we do provide warnings that "you may break X downstream workspaces" and allow you to contact those workspace owners. This isn't something we can easily do with the public registry simply due to the data model and not wanting to have everyone's usage information stored (we don't store or have access to any of that right now nor do we have plans, don't worry). But the private registry will be doing all of that because the application has all that information already from Terraform Enterprise.
And beyond all of this: Terraform 0.11 (due for release late this year and we talked about it during the HashiConf keynote briefly) is bringing along huge improvements to the module system in the core of Terraform which will make it much easier and more robust to work with modules.
Lots of stuff! But I hope this helps answer some of the questions for you and others.
(I'll add an edit since after posting this I saw some of your other comments: we are planning some major changes around HCL, we've publicly said so on issues, and we expect those to start hitting around 0.11. There are definitely some mistakes we shipped in its original form and while that's unfortunate, but we've learned a lot and brought on some people very experienced on such topics and we plan to address these shortcomings over time.)
How is this any different from the various packaging mechanisms that exist for all the different programming languages out there? You make it sound like this is a new kind of problem that started with Docker.
When I add some dependency to the Maven pom of my legacy Java EE app, I'm exposed to exactly the same kind of risk.
In short: research any dependency you pull in, whether it is a library for your favorite language or a module for your infrastructure automation tool of choice.
It's a difference-of-kind problem when you're working on infrastructure. You unwind a bad dependency in your app by removing the dependency. You unwind a bad dependency in your infrastructure by destroying large whacks of it.
Whether it's true or not, I think there's a perception of a more cavalier attitude around Docker that things will "just work" if I make my whole build system and production deployment depend on a random lady's docker file.
Also, a lot of developers don't spend the time they ought to learning (or hire the people who already know) how a proper tech stack works to be resilient.
Tools like Docker and Terraform make it seem magic, which it's not. PyPI, Gems and NPM don't make Python, Ruby and Javascript seem like magic, as most devs are deep enough in the code to see a terrible library before building their app around it.
Agreed, and the perception here matters. The cult of the five minute demo is real and people pick tools outside of their areas of expertise based on "well, it looked okay and there are blogs about it". The likelihood of bad stuff (whether bad through ignorance, malice, or lack of maintenance) is much, much higher when using these artifacts that are opaque not (just) in terms of source--because, yeah, most people won't audit it--but in terms of functionality (because the adoptee doesn't know the topic well enough to know what's going on).
You solve infrastructure problems with code, but infrastructure is not code and infrastructure requires a depth of understanding nobody's going to make disappear based on "magic." As you go downstack, the damage you can do increases. Depending on J. Random Dockerfile is problematic, as you're exposing your compute surface to the predations of a malicious update. Depending on J. Random Terraform Module opens you up to way deeper issues. These issues can only be solved by really understanding your stack, and five-minute-demo culture does not value understanding.
This is unfortunately necessary as code eats the world, and the number of programmers, tools and ecosystems proliferates. Popularity is necessary for infrastructure tools to get enough eyeballs and resources to achieve stability. It's not enough for experts with decades of experience to design really solid but arcane tools, because the flood of new startups with young engineers coming in saying "do we really need that?" will overwhelm than faster than all the failures can be pointed out.
Ultimately, the quality of the tools is orthogonal to the presentation. My perception is Hashicorp has done much better in this regard than Docker, so there's that at least. Only time will tell how it really shapes up over time, but I believe Hashicorp tools are seeing massive uptake for more reasons than just good marketing.
I really like Terraform but haven't been a fan of the existing way of bringing external code into my terraform scripts, via some strange go-like "refer to this github" repo mechanism.
Hopefully this makes them package able artifacts. Looking forward to watching the keynote and other videos from Hashiconf.