[xyzzy_plugh]

I'm not sure what you're on about. Operators don't need to run in cluster at all. And even then, they can absolutely run as containers. And as far as permissions go, that's up to you. They're just regular service accounts.

[remram]

Custom resource definitions are global to the cluster, and operators are usually written to watch those resources in the whole cluster.

You run the operator itself in a pod, but you can only have one of them running at a time, hence they are global. You can run as many Alpine/Ubuntu/... containers as you want, and as many ElasticSearch or PostgreSQL as you want, in isolation, that's the beauty of containers. But you can only run one SpiceDB operator or one ElasticSearch operator, on the whole cluster. We're back to square one.

[evancordell]

I don't think it's quite as bad as you're suggesting. Operators are control-plane, your other examples are data-plane. You can also only run one Deployment controller in a kube cluster, but many Deployments. And you can run many controllers for a single CRD, see the Gateway APIs for example, but it's not as common and is more work.

That said, I do think the direction that Kube has gone with CRDs, by leaning into making them more like built-in apis instead of allowing them to be differentiated as an external extension point has limited what you can do with them.

When ThirdPartyResrouces just carved out a group and a name in the API it was simple to have multiple copies and multiple versions running together in a cluster. But that has become more difficult with time, especially with ConversionWebhooks - a feature that forces all controllers for an API to agree not just on a version, but on the specific ways versions convert into each other.