[remram]

The SpiceDB operator looks like a prime example of something that should have been a Helm Chart. Migrations can be run in the containers.

Operators are just the non-containerized daemons of the Kubernetes OS. We did all this work to run everything in neatly encapsulated containers, and then everyone wants to run stuff globally on the whole cluster. What's the point? Do we just containerize clusters and start over?

[evancordell]

I get the sentiment. We held off on building an operator until we felt there was actually value in doing so (for the most part, Deployments cover the operational needs pretty well).

Migrations can be run in containers (and they are, even with the operator), but it's actually a lot of work to run them at the right time, only once, with the right flags, in the right order, waiting for SpiceDB to reach a specific spot in phased migrations, etc.

Moving from v1.13.0 to v1.14.0 of SpiceDB requires a multi-phase migration to avoid downtime[0], as could any phased migration for any stateful workload. The operator will walk you through them correctly, without intervention. Users who aren't running on Kubernetes or aren't using the operator often have problems running these steps correctly.

The value is in this automation, but also in the API interface itself. RDS is just some automation and an API on top of EC2, and I think RDS has value over running postgres on EC2 myself directly.

As for helm charts, this is just my opinion, but I don't think they're a good way to distribute software to end users. The interface for a helm chart becomes polluted over time in the same way that most operator APIs become polluted over time, as more and more configuration is pulled up to the top. I think helm is better suited to managing configuration you write yourself to deploy on your own clusters (I realize I'm in the minority here).

[0]: https://github.com/authzed/spicedb/releases/tag/v1.14.0

[xyzzy_plugh]

I'm not sure what you're on about. Operators don't need to run in cluster at all. And even then, they can absolutely run as containers. And as far as permissions go, that's up to you. They're just regular service accounts.

[remram]

Custom resource definitions are global to the cluster, and operators are usually written to watch those resources in the whole cluster.

You run the operator itself in a pod, but you can only have one of them running at a time, hence they are global. You can run as many Alpine/Ubuntu/... containers as you want, and as many ElasticSearch or PostgreSQL as you want, in isolation, that's the beauty of containers. But you can only run one SpiceDB operator or one ElasticSearch operator, on the whole cluster. We're back to square one.

[evancordell]

I don't think it's quite as bad as you're suggesting. Operators are control-plane, your other examples are data-plane. You can also only run one Deployment controller in a kube cluster, but many Deployments. And you can run many controllers for a single CRD, see the Gateway APIs for example, but it's not as common and is more work.

That said, I do think the direction that Kube has gone with CRDs, by leaning into making them more like built-in apis instead of allowing them to be differentiated as an external extension point has limited what you can do with them.

When ThirdPartyResrouces just carved out a group and a name in the API it was simple to have multiple copies and multiple versions running together in a cluster. But that has become more difficult with time, especially with ConversionWebhooks - a feature that forces all controllers for an API to agree not just on a version, but on the specific ways versions convert into each other.