I have a cluster using terraform-hcloud-kube-hetzner [0] and I'm quite happy with it.
It uses MicroOS for rolling releases and auto-upgrades for less maintenance. I'm quite happy with it so far, and I like that it integrates with terraform.
I had never heard of MicroOS before, sounds interesting. Before building this tool I actually used both Terraform and Pulumi. My tool is a ton faster setting things up and only requires a small YAML config file. Would be able to give it a try? I am looking for feedback :)
I can test it. What would be the advantage if I use it in my tool instead of Ubuntu? At the end of the day with Kubernetes I don't think it makes much difference which OS you use. I picked Ubuntu because it's standard and servers with this image build very quickly with Hetzner Cloud.
CoreOS (and its spiritual successor Flatcar) are immutable, which gravely limits the silliness that can be done to them post-launch. We use now use Bottlerocket since we're on EKS, but it's even more locked down since to even get an interactive shell is some major hoopjumpery. I believe Talos goes even further and is completely devoid of a shell
So, yes, it absolutely matters which OS you use, of course depending on your threat model and tolerance for "no, you don't get to ssh onto a Node and do whateverthehell you want"
We use Pulumi at my current workplace and love it. Being able to use a real language (Typescript, in our case) allows us to create data structure abstractions to feed our Infrastructure-as-Code that make sense from a developer interaction point-of-view. Our abstractions are less leaky and more in-line with a well designed interface, ready for use by a team that doesn't need to know all the details. We can have developers write feature code and design services and just plug in simple details for new services in our stacks Yaml.
However there is a gotcha. There are nearly infinite ways to setup your Pulumi codebase and there is no real prescribed structure outside of "create stacks". This can lead to a lot of time spent in architecting / implementing / testing different structures and abstractions which are not really a thing with Terraform. This can take time and can lead to refactors if you are not extremely intentional with your design.
I used Terraform first and then switched to Pulumi. What I like of Pulumi is that you get to use a proper programming language rather than a DSL, so you have more control/power in a way.
It's up to you. You can configure it with one master or HA multiple masters (e.g.). If you set the instance count for the masters pool >1 then my tool will create a load balancer in front of the API server.
The tool simply requires a small YAML config file and creates and configures everything that you need to get a fully functional cluster in a few minutes. It creates all the infrastructure resources (servers etc), it deploys kubernetes on all the nodes, and then installs some software that allows you to provision both load balancers and persistent volumes. It also installs a controller that makes upgrading to a new version easier.
What the tool does not install is other apps, because this would be too opinionated and depends from cluster to cluster.
I can give you an example of one cluster than I use for personal stuff:
- 3 masters with 3 cores and 4 GB of RAM each
- load balancer for the masters (for the Kubernetes API)
- 3 workers with faster AMD CPUs and 4 cores/8GB of RAM
- around 200GB of storage in persistent volumes
I think it costs around 60-70 euros per month total! I can't remember the exact amount because I have many things with Hetzner and do tests creating and deleting stuff all the time so my bill is rather long lol.
Hetzner is very cheap, but I can voice for the performance and reliability as well. It's a very good provider and I highly recommend them. Now they even have 2 locations in the USA!
That’s cost-effective! I’ve been toying with the idea of running a personal distributed system that’s not serverless to just hack around and Hetzner seems approachable going by your experience.
Definitely. I have been using them for years and I still can't understand how they can offer such a good service at their prices, considering how much the competition charges.