[whalesalad]

Kubernetes is really epic and powerful if you actually take the time to understand it from first principles. Unfortunately people don't do this, and individuals without good networking/devops experience roll something half-baked out with a terrible deployment process, a mess of helm charts, etc... and it ends up being hated by everyone.

At FarmLogs (yc 12) we had a pretty righteous gitops (homegrown) kube platform running dozens of microservices. We would not have been able to move as quickly as we did and roll out so many different features without it. This was back when people had just started to adopt it. Mesos was still a contender (lmao). We were polyglot too - python/clojure mixture. Heck, we even ran an ancient climate model called APSIM that was built in c#/mono, required all kinds of ancient fortran dependencies etc and it worked like a charm on kube thanks to containers. We had dedicated internal load balancers behind our VPN for raw access to services and endpoints, like "microservice.internal.farmlogs.com" (this was before istio, fabric networks, all the incredible progress that exists now)

I recall Brendan Burns asking me to write up a blog post for the Kube blog about our success story, but unfortunately was so saddled with product dev work and managing the team that I never found time for it.

I will absolutely adopt K8s again one day (very soon) but you need to know how to harness its capabilities and deploy it correctly. Build your own Heroku that fits your business. Use the Kube API directly. It's really not hard. It gets hard due to all the crap in the ecosystem (helm, yaml files). Hitting API direct means no yaml =)

I am stoked to see Fly offering this.

[ekidd]

I have used a number of orchestrator platforms in production, including AWS ECS (Docker + custom orchestration), EKS (Kubernetes) and Google Cloud's Kubernetes.

I've also used Chef, custom RPM packages and classic Unix startup scripts. And I'm probably forgetting some.

And honestly? Kubernetes can be really great. Especially if you:

- Read enough to understand the split between pods/replication controllers/deployments, which is a bit unusual, and the fact that "services" are basically a name lookup system. This split is weird, but it's not that hard to figure out.

- Pay someone for a quality managed Kubernetes.

- Don't get clever with the networking overlays.

I especially like the way that Kubernetes allows me to deploy almost anything with a short YAML file, and the fact that I never need to worry about individual servers at all.

Now, I wouldn't use Kubernetes if I could get away with a "Heroku like" system. But for anything more complicated than that, Kubernetes can be pretty simple and reliable. Certainly I'd take Kubernetes over a really complex Terraform setup.

[whalesalad]

There is absolutely (still!!!!) room in the market for a PaaS built on top of Kube that is actually good, and is a hybrid of Heroku+Kube. More convention, less all the rope to hang yourself with, while still enabling advanced use cases with full control.

Even docker compose is too annoying for what most people need.

Most of our use cases are so simple. I need a (black box) container that exposes a TCP port. I want N of them. I want them load-balanced with a friendly name. I want them load balanced behind a friendly name with auth in front of it. I want centralized logs, based on these names. I want centralized stats, based on these names (not xyz-pod-232903209284390-dev but xyz aggregated). I want auto deploys based on a github repo. I want my releases tagged with the git short hash.

When someone cracks the nut of heroku+kube they will become the next billionares. This is why I think it is wise to try and enter this space, as Fly is doing.

The new stuff Microsoft has announced (named Radius, not to be confused with RADIUS network auth protocol) is the closes thing to this sort of solution I have been imagining for years: https://azure.microsoft.com/en-us/blog/the-microsoft-azure-i...

I want N http/tcp/php/python/node/insert-tool-here API or web servers. I want a best-practices RabbitMQ deployment, a best-practices Redis deployment, and a best-practices PostgreSQL deployment. For Redis, I do not care about state, just make it work. For Rabbit, I want 3+ nodes for HA. For PSQL, I do care about state so please use <insert EBS volume here> and do backups.

This is the same shit we have been reinventing for years over and over again and the recipes are all the same now. Radius is the best (at least on paper) attempt at unifying all this stuff.

I've been beating this drum for over 10 years now though, I keep thinking someone will figure it out and no one has. Maybe time to bite the bullet and just do the thing.