[ferdowsi]

From the application engineer side I'm not convinced that Kubernetes is particularly complex. I recently ramped up on it and found it liberating, frankly. Once I understood the basic concepts it was much more sensible than staring at a mountain of bespoke Ansible scripts operating on components I could barely see or understand.

I can't speak to the SRE side; I can imagine the complexity there. But are these challenges greater than maintaining, observing, and modifying a Rube Goldberg machine of managed AWS services?

[gizdan]

I have been on both sides of the stick.

At a previous place we set up a cluster on AWS. This was before EKS. We started out with kops initially but later used the generated CFN yaml. It was not an easy feat. There was a lot of gotchas, moving pieces and much more. All of these moving pieces had their own gotchas. Plus lots of competition in the area with not a lot of comparisons since it was early days. Many things were not fully stable. A lot of issues. We got there in the end, but it wasn't easy.

On the flip side, we were able to onboard people with their services in days, not weeks (previously the company ran their own datacentres). Teams were allowed to go to AWS directly, or go to our K8s cluster. I was able to observe the lead time of 3 teams, 2 chose k8s, 1 chose AWS. Those going to K8s were able to get their prod environment running within a week. The other team took a month to do their dev environment. All three teams deployed a single stateless service.

This is obviously anecdotal, but I was really impressed with the user friendliness of kubernetes for the consumers.

Nowadays, 4 years later at my current company, it's a different story with every major provider managing the clusters for you. At my current company we haven't taken the jump yet, so unfortunately I can't fully compare, but the little I've played with EKS, it's as easy as simple crud operations.

[sleepybrett]

I'll back up your anecdote. We had a lot of teams fretting about how long it would take to utilize kubernetes. After a short sit down with one of the kubernetes ops team members (even juniors) they were up in running and thanking us for all the time they would save managing ec2 patching etc.

[bostonsre]

There is a large amount of stuff to learn about how to do stuff the kubernetes way but yea, it's just a tool to run isolated processes across a fleet of servers. You can make a mess in ec2 and in kubernetes and I don't see much if any additional complexity to costs associated to running kubernetes. It is friggin gorgeous and elegant when comparing it to something like hadoop. Spark on kubernetes has been a dream compared to writing automation that used cloudera or ambari hadoop management layers.

[kinghajj]

"Any sufficiently complicated deployment and application management system contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Kubernetes."

[cat199]

... including kubernetes itself

[clipradiowallet]

> I can't speak to the SRE side

In all seriousness, what other side is there? The SRE's role is to make sure you never encounter kubernetes. eg... you have a git repo and some branches - if you push to them, deployments magically happen. As a non-SRE, what parts of kubernetes would you actually be touching or interacting with?

[ferdowsi]

> The SRE's role is to make sure you never encounter kubernetes

In my org feature development teams own the K8S configuration and uptime for their own workloads/services.

The SRE team owns the shared Kubernetes platform itself and the related infra/config (cluster-level config, RBAC, the CI/CD infra, certificate tooling, the secrets infrastructure, the observability pipelines, etc).

[nycdotnet]

bouncing pods and occasionally tailing logs in the case of startup problems is about it

[jansommer]

Had the exact same experience. Was planning to use Ansible until it became obvious that Kubernetes was a perfect match, and it was fairly easy to learn the basics. Getting it to run correctly within our VPC was a pain for the developer doing that part though.

[KurtMueller]

I have a basic Ruby on Rails app up on Kube but with no authentication and no SSL certificate. What should I learn next to add these things? How did you learn these things?

[fimbulvetr]

Maybe using cert-manager & ingress-nginx to front the service with SSL. Then your ingress-nginx will front your ruby app and decorate it with ssl. Please note, and I am embarrassed it took me so long to figure out - there are two nginx ingress projects - "nginx-ingress" and "ingress-nginx"! I would strongly recommend using the k8s official one, which is ingress-nginx. See: https://www.digitalocean.com/community/tutorials/how-to-set-...

This will help you learn the ingress pattern. After that, I would suggest exploring ways to tack a sidecar on (log aggregator, etc) - my impression is you are just looking for things to learn, I wouldn't normally suggest doing this just to do it.

Alternatively, you could try exploring putting grafana/prometheus in, though this can be a big bite for someone learning, so I would recommend learning/comprehending sidecars/ingress, etc as they are some of the building blocks for k8s.

Edit: I see there's another comment for traefik for ingress - that's fine too, it's the concept that matters, not your particular choice. If you have a lot of trouble implementing one, try the other one, things you learn in your journey will help quite a bit.

[scns]

Use Traefik as your Ingress, done.

[scns]

Correcton: for TLS, Auth still has tbd.

Cant edit since on mobile

[dcow]

Just refresh the page the edit button should reappear on mobile.

[scns]

Thanks for the tip. On mobile means using Materialistic for me, no way to edit there.

[twalla]

From the SRE side: if you need bespoke fancy shit (we write a lot of operators to remove toil from running things we'd typically do with ansible/manually), the kubernetes API, CRDs and the operator/controller pattern are a joy to develop against for the most part.

[throwawayboise]

My only experience with k8s is trying to run it in-house on a cluster of servers. I could never get it to work. Ansible makes much more sense to me.

[clipradiowallet]

How do k8s and Ansible overlap in your world? I personally haven't encountered a situation where one can serve as a substitute for the other.