[tenfourty]

[OP here] It feels bizarre saying this, having spent so much of my life advocating for and selling a distribution of Kubernetes and consulting services to help folks get the most of out it, but here goes! YOU probably shouldn't use Kubernetes and a bunch of other "cool" things for your product.

Most folks building software at startups and scale-ups should avoid Kubernetes and other premature optimisations. If your company uses Kubernetes, you are likely expending energy on something that doesn't take you towards your mission. You have probably fallen into the trap of premature optimisation.

Please don't take this post to be only aimed against Kubernetes. It is not. I am directing this post at every possible bit of premature optimisation engineers make in the course of building software.

[xyzzy123]

Doesn't it depend what staff you have? I would have agreed with this in 2018 but the world has moved on.

If you have a bunch of people who know it, you can deploy a cluster into gcloud or aws with a few clicks or lines of IaC.

Would I recommend a startup team learn kube while trying to ship a product? No.

Would I think it's a red flag if a team who already know it choose it as their preferred platform? Also no.

[ramraj07]

Even if they know it, unless they absolutely need to implement it, why would you waste such valuable resources on of all things infra? (Unless your product IS infra). No engineer I know who’s smart enough to effortlessly deploy k8s on their own would want to do that as the job. There’s a million other interesting things (hopefully?) that they can be doing.

[dinkleberg]

Not at all. If you know Kubernetes well and your needs are fairly simple it takes no time at all.

On a project a couple of years ago my cofounder and I opted to use Kubernetes after running into constant frustration after following the advice to keep it as simple as possible and just use VMs.

Kubernetes makes many complicated things very easy and lets you stop worrying about your infra (provided you’re using a managed provider).

On our next project we used a PaaS offering because we had a bunch of credits for it and it was painful in comparison to what we had with Kubernetes. It was way more expensive (if we didn’t have credits), the deployments were slow, and we had less flexibility.

Kubernetes isn’t perfect, far from it. But for those who know it well it is usually a good option.

[xyzzy123]

Often to get access to things that integrate well with kubernetes.

Personally I don't see too much difference between kube yamls and systemd units and cloudformation or whatever and the "cluster maintenance" itself is not the burden it used to be if you stay on the "paved road" provided by your cloud.

[zelphirkalt]

What about maintenance of that cluster (of possibly only 1 machine) though?

[hadlock]

You can get away without an orchestrator right up until about when your ARR hits $10mm

Hiring people with k8s experience in 2022 is not a difficult task, it's not an obscure technology anymore, and when your initial crop of devops decides to leave, the new guys coming in can scan your deployments and namespaces and pretty much hit the ground running within 48-72 hours. That's a big, important part of running a business. Being able to hire people with the right skill set, and being able to keep the lights on.

Bespoke systems built on top of EC2 require time, extra documentation and a steep learning curve, not to mention the fact that the bespoke system probably isn't getting security audits or being built to modern standards or best practices, tooling isn't being kept up to date. I can build a mechanical wristwatch in my garage, but if I had full access to the Rolex factory floor for free, I'd probably take that option.

[pjmlp]

So much for theory.

I just came out of a project where Kubernetes performance issues involved wild guess and blind tuning until the so called experts actually found out why the cloud cluster was behaving strangely, including support from Cloud vendor.

And good luck making sense of all the YAML spaghetti available for bootstrapping the whole cluster from scratch.

72 hours? They better be 10x DevOps team.

[secondcoming]

This is our world right now. A kubernetes cronjob sometimes fails and we have no idea why.

[hadlock]

If your devops guys are struggling, you are hiring the wrong devops folks, or at the wrong end of the pay band. Most devops guys I know are paid in the same band as a senior developer.

[pjmlp]

Sure, it is like bug free C code. It is only a matter of having the top of the cream. Pity there aren't enough of them in the world, including on cloud vendor support team.

[mschuster91]

Running a self-hosted Kubernetes is indeed ... questionable, but a managed Kubernetes? That's a pretty sane thing to do IMO. The alternatives are running your Docker containers either manually on some EC2 or other bare-metal server which is a nightmare to do deployments, or using something like Elastic Beanstalk which is even worse.

For me at least, Kubernetes has become something like an universal standard: if you're already running Docker or other containerization, "using Kubernetes" is nothing more than a clear, "single source of truth" documentation on what infrastructure your application expects. Certainly beats that Confluence document from 2018, last updated 2019, on how the production server is set up that doesn't even match up with what was reality in 2020 much less today.

[k8sToGo]

A managed cluster is not as much managed as you would think. Still have to configure and install a lot of things yourself.

[dj_mc_merlin]

Of course, managed means you don't take care of say, creating your own X.509 CA and issuing certificates, tending to their expiry, installing Tiller, setting up pod networking etc. etc.

All of these much more annoying and harder than `helm update --install`ing some charts to your own cluster.

[grumpy-de-sre]

To be fair installing tiller hasn't been a thing for years. And pretty much everyone using cert-manager and lets encrypt which makes the whole X.509 story pretty much a no brainer.

[k8sToGo]

That is true. But if you compare it to, for example, managed web hosting, then it is a very different managed experience.

[subhajeet2107]

What about standardisation that comes with using a framework like kubernetes , while not using k8s you end up with adhoc deployment methods, clunky work arounds of handling networking, policies, secrets etc, with Kubernetes or even ECS it signals that the team or the developer is looking to use fixed set of rules for infrastructure, also k8s scales well even for smaller apps

[adastra22]

Seriously, I use k8s for the same reason I use docker: it's a standard language for deployment. Yeah I could do the same stuff manually for a small project.. but why?

[p_l]

Last year I tried to "simplify" by doing things mostly the old way (though we used containers for a bunch of things later on).

The experience pushed me to decide "never again" and to plonk k3s or at least something like podman from the start on the server.