[mycentstoo]

As an SRE who’s moved between a few companies recently, I’ve noticed a trend towards simpler infrastructure options rather than k8s. Most places I interviewed at used AWS with just fleets of EC2s managed through ASG and instance refresh.

Deployments are no downtime. It’s pretty nice and if you want to run with docker on the host instances you can but you can also just install the application using systemd.

[jauntywundrkind]

How do you build ec2's? Funny thing is, everyone will have their own answer. Few will look alike. It's maybe simpler, but also bespoke.

There's different definitions of simpler. I've rarely felt like folk's have a great grasp when they cobble together a bunch of options. Things that maybe one engineer started as a simple project 5 years ago grow and sprawl. They rarely stay simple.

What this team gets out of the box here is not to be trivialized. You havent started talking to the ci/cd that ArgoCD deploys. How are load balancers setup & managed, and at what price (Traeffik here is basically free)? Well managed Postgres with backups & replication, Lets Encrypt... handled. Is it really simpler to keep DIY'ing bits and bobs? Are you confident your simple start will work reliably & won't need improvement? What is better about your pile "simple" bespoke solutions, and why not have a platform that integrates varying concerns with consistent management patterns & autonomic operators?

[mycentstoo]

EC2s are provisioned with Terraform or if you want to bypass that, click-ops. Nothing custom is needed. You can define a small shell script that is known as userdata that runs when the EC2 boots. That's standard and used everywhere that EC2s are used.

If you want more customization - again not necessary - you can build via Packer. The thing is, you have to do that anyways with Docker or whatever your container runtime happens to be. Docker images need patches as well if you install dependencies like openssl, libpql, etc.

ArgoCD - ArgoCD is great but it's something that has to be separate managed and comes with a database, UI layer, etc. and it's own set of permissions. The whole point of ArgoCD is to have deployments specifically around GitOps for container-based applications. Unnecessary if you don't have K8s.

Load balancers are simpler with ALBs and route directly to the auto-scaling groups. Traeffik is something that you need to manage. Also, K8s typically requires the use of external load balancers to work in the cloud as it is which means there's a layer of complexity that was introduced. Typically, K8s through the External Load Balancer Controller will provision the LB and attach targets. So it adds complexity. Traeffik isn't free, you are paying for the compute in your cluster which has the overhead of the control plane.

Let's Encrypt is another layer of complexity that has to be managed. AWS has ACM which can dynamically create and renew your certifications without you having to do anything.

K8s as a cluster needs to be managed and introduces significant breaking API changes across minor versions (1.23-1.24, etc.). Postgres managed as a stateful application also breaks some of the benefits of K8s in that it needs to be managed as a StatefulSet rather than as a ReplicaSet which means you likely need different deployment mechanisms to avoid destroying it accidentally.

Ironically, this architecture doesn't need any customizations at all. You can provision it with a few modules in Terraform or you can click ops the entire thing. K8s on the other hand, requires pretty constant attention. You've now introduced the management of the cluster, the building of the image, management of ArgoCD, Traeffik as well as external load balancers.

Basically: - Route53 for domain management

- ACM for cert management

- EC2s for LBs

- ASGs to manage EC2s

- Small shell script within userdata to configure the host vm

- RDS if you need a database

Avoiding the operational and actual costs of K8s.

I think that there are compelling reasons to use K8s:

- Lots of smaller services that need to interface with each other

- Product is deployed to multiple clouds or in a hybrid setting

- Infrastructure team has deep expertise in K8s

[icedchai]

Yep. What you describe is how we were doing things 10 years ago!

[jauntywundrkind]

I think you'd spend longer just figuring out terraform, step 1, than you would doing every single piece of this blog post.

[secondcoming]

This is what we do for GCE. Our CI builds a system image and then that's deployed across our datacentres. Worked perfectly for years, but still people want to switch to kubernetes. What can you do?

[mycentstoo]

Oh definitely that happens and from my experience I don’t think there’s a lot that can be done unless there’s strong faith in engineering leadership to push back on the complexity.