[arkh]

Tried Pulumi thinking "it's gonna abstract all the k8s specifics". Welp no, still need to know and understand K8s so I still don't see the value from those kind of tools. In which case why not use something like Pkl to generate my yaml from some sensible code-like structures?

[katdork]

kubernetes is very complex and therefore any abstraction which completely glosses over the way the underlying systems work would make it very hard to avoid leaking or a bad abstraction to begin with.

the complexity in one way or another must be preserved within the abstraction (in all likelihood) or you will have cases you cannot create in that layer or breakages which now have the total complexity of both the abstraction itself AND kubernetes itself required to fix.

i would not say IaC is going to provide you a magic solution to learning k8s, although the value in using IaC (e.g. Argo CD / Flux CD + Kustomize + ...) in K8s land is that you are no longer imperatively managing your cluster resources and therefore can keep them within a repository, managed like code. the point of the solution is not to make it easier for newcomers, but to make it easier to have teams manage and work together on an established cluster for deployments, ...

in the case of Pulumi, you leverage the single language with typechecking instead of relying upon K8s flavoured YAML, which is itself beneficial in many ways (since you can use your regular developer tooling)

wrt pkl, pretending K8s manifest structure underneath does not help because you will need to know how the keys within a manifest interact with the underlying system regardless, especially to understand functionality, e.g. node selectors, taints and tolerations, node affinity, ...

i prior managed a terraform-based deployment of several k8s clusters and it still required knowledge of those keys and values, alongside knowledge of the underlying resource types.

without those you can't implement things like GPU-based node selection for jobs which require a GPU, ...