[cs02rm0]

I've hardly looked at k8s, it looked like a pain to get a docker-compose file running in it the last time I looked. I think there was a tool to translate it to something kubernetes could deal with.

That said, after the last four months I've had of trying to get Route53, API Gateway, Lambdas, ECS, VPCs, Cognito, DynamoDB, S3, CloudFront, etc, etc running in CloudFormation I really can't imagine it's worse under any slightly involved scenario.

[jpgvm]

Translating something from docker-compose into k8s isn't generally hard. But yeah CFN is a nightmare, I wrote a bunch of tools to try make it suck less and contributed to stuff like cfndsl and I could still never get it to the point that I could have developers use it without an abstraction layer.

k8s definitely helps with the "all my apps stuff in one place that is easy to see". There are some pitfalls to avoid though. Don't use helm, it's bad for your health. Avoid deploying your own k8s cluster unless you really need to, just use GKE. Avoid custom resource definitions unless they are well supported, migrating off them can be hard - prefer tools that look at annotations etc (like external-dns, cert-manager and friends).

Of course things get difficult when you need statefulsets to run things like Kafka/ZK and friends but it's definitely possible and they run well once setup.

In my mind k8s is the only option right now that doesn't result in man-years being wasted on pointless AWS bullshit.

[sklarsa]

Why not use helm? I'm looking into spinning up (and eventually productionizing) a k8s cluster at my job and I was leaning towards using helm since some pieces that I was thinking about using are installed via helm charts (https://github.com/kubernetes/ingress-nginx for example)

[solatic]

If you use third-party Helm charts, you eventually need to add onto the generated objects in a way which the chart doesn't support, and then you're up a creek without a paddle. This is precisely the use case which Kustomize tries to fix and it was the only real strength which Helm had to begin with (i.e. the ease of installing third-party software on your cluster).

In the meantime, because you must install Helm into every namespace in your cluster into which you desire to install charts, it's a massive resource hog and security risk. Charts themselves also need to be hosted somewhere, so you end up needing to install Chartmuseum, Harbor, or Artifactory (if you didn't have Artifactory already), and they have their own operational costs.

[navd]

I thought it would be useful to add that you can also generate k8s manifests from a helm chart using the `helm template` command.

I'm in the same boat where I avoid helm if at all possible.

[alexandre_m]

That only works for local checked out copy and not remote repo URL, unfortunately.

[alexandre_m]

Something not pretty, but that works well for me, is using yarn to manage dependencies on upstream Helm chart repositories, then use kustomize to override certain things if required.

That checkouts a local copy locked at a specific version, which you can bump easily, and allows overriding the template definitions on your side.

No need for additional infrastructure.

[fimbulvetr]

They're really not as bad as you've made it sound. I've been working with CF for 1.5 years, and I've done everything you've listed above (and more) with deep integrations into each other, and more. The most difficult was cognito, but that's because of their naming choices. Second most was ECS, but that's because of the time needed to flesh out and reason about everything.

There are plenty of examples on github/gitlab for most of the services you mentioned. Like most programming languages, you need to separated the wheat from the chaffe; 99% of the CF publicly available sucks. Hit me up if you need help.

[jpgvm]

I used it heavily for over 5 years. I know how to use it but that isn't the problem.

The problem is I can't give it directly to developers. I don't want to be constantly writing/maintaining abstraction layers over CF to make it half decent. You may as well use Terraform at that layer of abstraction because at least it frees you of some lock-in and has more sane state tracking (which too is pretty abysmal but we aren't talking about a high bar when comparing to CFN) but I digress.

You seem to be missing the point here, it's not that you can't do things with CFN, you can. The point is k8s allows me to administrate a system that puts all of the machinery behind a nice unified API that all of my developers can consume without requiring an SRE to make every change and be there to hand hold them through setting up a new service.

It may be more complex but that complexity quickly pays off with the amount of stuff now solved in a self service manner. Sure some stuff needs to get much better, there are poorly understood primitives like Service and Ingress that could do with better docs to help explain to devs when you should use one and why.

Overall things are much better than the "AWS as an abstraction" way where everyone is clamouring to use some half baked AWS garbage service rather than just doing the old tried and true thing (screw Kinesis for example).

[dankohn1]

Yes, you're thinking of https://github.com/kubernetes/kompose

[merb]

you do not need docker-compose. you can just use k3s/minikube or k3d. so you can actually run your whole dev environment directly in kubernetes.