[imstil3earning]

> To move to Kubernetes, an organization needs a full engineering team just to keep the Kubernetes clusters running

Perhaps my team runs a simpler cluster, but we have been running a Kubernetes cluster for 2+ years as a team of 2 and it has been nothing less than worth it

The way the author describes the costs of moving to Kubernetes makes me think that they don't have the experience with Kubernetes to actually realize the major benefits over the initial costs

[heydonovan]

Yes! Was going to say the same. Kubernetes is far easier to learn than some random bespoke setup. After reading the article, it just sounds like they reinvented the wheel but made it AWS specific for some reason.

Was brought on as a consultant years ago and found their bespoke setup of random ec2 instances and lambda scripts to be far more difficult to understand than just spinning up a managed Kubernetes cluster and having a generic interface to deploy the application, as well as monitoring, logging, metrics, etc.

[cortesoft]

> Kubernetes is far easier to learn than some random bespoke setup

This, to me, is the biggest advantage of kubernetes. Yes, you can do all the things yourself, with your own custom solution that does everything you need to do just as well as kubernetes.

What kubernetes gives you is a shared way of doing things. By using a common method, you can easily integrate different services together, as well as onboard new hires easily.

Using something ubiquitous like kubernetes helps both your code onboarding and your people onboarding.

[throwaway894345]

Also not having to debug EC2 lifecycle issues is really nice. And you don't have to deal with all of the garbage of setting up ec2 instances (configuring SSH, process management, configuration management, log exfiltration, monitoring, infra as code, nginx, albs, etc).

[zinclozenge]

Well, that and also a "nice" programmatic interface.

[ldiracdelta]

In addition to ease, why would I, as a s/w engr, want to invest in learning your weird stack instead of learning or using an equivalent tech stack that is actually marketable? Learning a technology can be a huge investment. Why would I want to invest in a technology with not much future and not much ability to translate into another position should this gig go south.

[throwaway894345]

Note that Kubernetes skills are _hot_ right now. 6 months of Kubernetes experience is easily an extra $20k in salary.

[emodendroket]

Well, they say they're using CloudFormation and Terraform. Those are also fairly standard, popular tools, aren't they?

[cat199]

because you then know how to apply technology instead of a technology ?

[ldiracdelta]

So, I should be OK to use my most precious asset -- time -- on an investment in a dead-end technology, because that investment could possibly, maybe translate (in some reduced way) to me being able to use a different technology later on that is not a dead-end technology? How about I just invest directly in technologies that have a payback and not try to cram the entirety of all software engineering systems into my brain. I already have an unending amount of things and systems to learn in s/w, why shouldn't I prioritize the learning process towards investments that have the best paybacks.

[0xbadcafebee]

Kubernetes is literally more complicated than any other similar system in existence. It is so by definition, because it builds in more complexity than any other system has. But it also lacks the features you often need, so you have to learn even more to be able to get it to do what you want. You could write several books on running K8s in production.

[xtracto]

I was doing research to move set up some new system into scalable cloud infrastructure. My first option was K8s (EKS) and second was plain ECS+Fargate. Ramping up in K8s was so convoluted and painful that I decided to follow up with ECS. That has been quite straightforward.

[nitrogen]

My experiences with k8s have led me to never propose k8s in a new project. The k8s instances I have seen were bureaucratic and technical nightmares of unnecessary complexity. It doesn't provide a common ground for deployments because everyone will use and configure it differently.

[marcinzm]

>It doesn't provide a common ground for deployments because everyone will use and configure it differently.

Helm charts are used by 99% of the open source projects I've seen that run on top of Kubernetes. They are all written in a similar style so transferring settings between them is fairly easy. Helm will even create a barebones chart for you automatically in the common style.

[0xbadcafebee]

A helm chart is not a complete deployment system, it's just a wrapper around kubectl. Neither provides everything you need to completely manage deploys.

[zaat]

But you can't write books on running Linux in production, or Apache, or Windows, or Oracle, or... since the book shelves are (not literally but metaphorically) too crowded for yet another one on the subject

[pilotpilot]

What do you mean?

You used to work with Ably?

[Fiahil]

Exactly, plus all major cloud provider will happily host a kubernetes cluster for you if you ask them with your money. In a previous project, we managed more than 20 clusters on Azure (AKS) and AWS (EC2) with 4 people. The only tool that made this possible was kubernetes.

I've been running my own kubernetes cluster on a Raspberry pi. Does my cat count as an engineering team?

[throwawayboise]

I feel like I must be stupid because I've tried several times to set up k8s on in-house hardware and it has never worked. I've tried several different "recipes" and the networking never works right. IDK if it's because things are changing so fast that the popular "recipes" are already outdated, or if they are being published without actually testing them, but it's left a bad taste in my mouth.

I'm sure the big cloud providers make it easy for end users to use, but that doesn't help me.

[Fiahil]

You can start with K3s: https://k3s.io/

The networking part is always the most challenging. Everything between your router and your kubernetes cluster should still be routed and firewalled manually. However, if you can live with your home router firewall and a simple port mapping to your machines/cluster, then routing the traffic and setting up the cluster should be relatively painless.

[alanwreath]

This^^ I started out with kubespray because I was familiar with Ansible (I even contributed a very small bug fix), BUT k3s is just so awesome and out of your way. It’s not only easy to install but easy to remove.

[Okkef]

Don't run your own cluster, unless you do it to learn. If you're in the cloud, just use a managed instance and focus on building software.

[foobiekr]

How do people who do this debug issues in the k8s layer?

Or is this just the Jenga Model of Modern Software Development in action?

[GauntletWizard]

There's definitely a lot of Jenga going on here, but on the other hand: Kubernetes, when set up, has some very simple constraints that work. You don't often need to touch things at that layer; They become invisible, and when you're dealing with a cloud provider's hosted K8s, you don't get to touch them directly anyway.

K8s was a lot more simple earlier on. It's actually dying from adoption: There's a ton of dumb ideas bolted on top, that have become "standard" and "supported" because of demands from bad customers. The core is very clean, though, and you rarely need to interact with that.

[remram]

How does Ably deal with issues in the EC2 layer?

[foobiekr]

That's one of those great questions about AWS. We actually have had to contact AWS on multiple occasions about EC2 layer issues, and each time I was thankful that a VM construct is very simple, comparatively, to reason about.

[mycall]

Little known but latest VMware Fusion and VMware Workstation comes with kubernetes OOTB (vctl, kind). It has never been easier to start up a cluster.

[ofrzeta]

I don't know what specific problems you had with networking but I found using Kubespray an easy way to setup a cluster on different clouds.

[sandebert]

Sounds like a useless use of cat.

[theptip]

This comes up a lot around here; many people look at “Kubernetes the hard way” and think they need to run their own cluster.

Just use GKE or whatever the equivalent managed offering is for your cloud provider. A couple of clicks in the ui and you have a cluster. Its really easy to run, I rarely have to look at the cluster at all.

[cassianoleal]

I've been on a team of 6-8 running relatively large-scale clusters (2 big ones with lower hundreds of workloads through tens of namespaces/tenants, plus a couple smaller ones). To "keep the Kubernetes clusters running" is an afterthought at most. I mostly agree with your comment.

[RamRodification]

Do/Did you use some specific method or orchestration product to set it up?

[cassianoleal]

We use GKE but there's nothing stopping most organisations to do the same.

It doesn't bear much weight though, and I've had experience with other toolings largely to the same effect.

I agree with the sibling comment about upgrades - this is IMO where GKE really shines as a cluster upgrade is mostly a button-pressing exercise.

[Sahbak]

Not OP, but similar workloads, team of 5 We use kops to deplot and manage the clusters. The hardest part is updating, due to the workloads running on it. Other than that, little to no problems with kube itself.

[maxbreaker]

We used to run a very well run Kubernetes clusters with a single devops, then two. If you have senior devops engineers who know what they are doing and have the opportunity to greenfield it's really not that bad. We now have 4 devops engineers.

At the end of the day it's not necessarily Kubernetes that's difficult/time consuming, it's all the context around it.

How do you monitor/observability? Security? What's your CI/CD to deploy everything? How do you fully automate everything in reproducible ways within the k8s clusters, and outside?

I've been doing infrastructure management / devops for well over 23 years. Built & managed datacenters running tens of thousands of servers.

Kubernetes and AWS have made my life easier. What's really important is having the right senior people fully aware of the latest practices to get started. Greenfield now is easier than ever.

[marcinzm]

Same. We've been running EKS at my last two companies using Terraform without issues and very small teams. There's a learning curve for sure but the ongoing maintenance is pretty minimal. I suspect going with GKE would have a smaller learning curve.

[arwhatever]

Genuine question because I honestly have no idea: how much additional learning curve is added by the average cloud provider to create a Kubernetes hosting context/setup?

Knowing nothing about K, I’m constantly wondering how it could be simpler than dumping a binary into a PaaS (web) hosting context and letting it do its thing. I’m interested to learn.

[miraculixx]

I am running 5 k8s clusters on behalf of customers on 3 different cloud providers.

By myself.