[evancordell]

This is a common frustration of mine as well!

In the latest release of the spicedb-operator[0], I added a feature that allows users to specify arbitrary patches over operator-managed resources directly in the API (examples in the link).

There are some other projects like Kyverno and Gatekeeper that try to do this generically with mutating webhooks, but embedding a `patches` API into the operator itself gives the operator a chance to ensure the changes are within some reasonable guardrails.

[0]: https://github.com/authzed/spicedb-operator/releases/tag/v1....

[ojhughes]

Adding the patch api is neat! I’ve solved this in the past by embedding the entire PodSpec etc into the CRD

[remram]

Did you call your CRD "Deployment"?

[ojhughes]

It’s a common pattern [1]. Other than the GP patch example, how else can users override the child objects your controller creates?

1 https://github.com/prometheus-operator/prometheus-operator/b...

[remram]

I'm not saying they shouldn't customize their PodSpec, I'm saying if they do, your controller is exactly the same as the built-in Deployment controller, running user-provided pods. Why build a copy of it?

[ojhughes]

Because it’s an aggregate of a number of different resources types and custom logic not exactly the same as deployment.

I can see you don’t like operators and that’s fine. Many folks find it’s a useful pattern to follow

[sklarsa]

I might have to borrow that! Very clever

[remram]

The SpiceDB operator looks like a prime example of something that should have been a Helm Chart. Migrations can be run in the containers.

Operators are just the non-containerized daemons of the Kubernetes OS. We did all this work to run everything in neatly encapsulated containers, and then everyone wants to run stuff globally on the whole cluster. What's the point? Do we just containerize clusters and start over?

[evancordell]

I get the sentiment. We held off on building an operator until we felt there was actually value in doing so (for the most part, Deployments cover the operational needs pretty well).

Migrations can be run in containers (and they are, even with the operator), but it's actually a lot of work to run them at the right time, only once, with the right flags, in the right order, waiting for SpiceDB to reach a specific spot in phased migrations, etc.

Moving from v1.13.0 to v1.14.0 of SpiceDB requires a multi-phase migration to avoid downtime[0], as could any phased migration for any stateful workload. The operator will walk you through them correctly, without intervention. Users who aren't running on Kubernetes or aren't using the operator often have problems running these steps correctly.

The value is in this automation, but also in the API interface itself. RDS is just some automation and an API on top of EC2, and I think RDS has value over running postgres on EC2 myself directly.

As for helm charts, this is just my opinion, but I don't think they're a good way to distribute software to end users. The interface for a helm chart becomes polluted over time in the same way that most operator APIs become polluted over time, as more and more configuration is pulled up to the top. I think helm is better suited to managing configuration you write yourself to deploy on your own clusters (I realize I'm in the minority here).

[0]: https://github.com/authzed/spicedb/releases/tag/v1.14.0

[xyzzy_plugh]

I'm not sure what you're on about. Operators don't need to run in cluster at all. And even then, they can absolutely run as containers. And as far as permissions go, that's up to you. They're just regular service accounts.

[remram]

Custom resource definitions are global to the cluster, and operators are usually written to watch those resources in the whole cluster.

You run the operator itself in a pod, but you can only have one of them running at a time, hence they are global. You can run as many Alpine/Ubuntu/... containers as you want, and as many ElasticSearch or PostgreSQL as you want, in isolation, that's the beauty of containers. But you can only run one SpiceDB operator or one ElasticSearch operator, on the whole cluster. We're back to square one.

[evancordell]

I don't think it's quite as bad as you're suggesting. Operators are control-plane, your other examples are data-plane. You can also only run one Deployment controller in a kube cluster, but many Deployments. And you can run many controllers for a single CRD, see the Gateway APIs for example, but it's not as common and is more work.

That said, I do think the direction that Kube has gone with CRDs, by leaning into making them more like built-in apis instead of allowing them to be differentiated as an external extension point has limited what you can do with them.

When ThirdPartyResrouces just carved out a group and a name in the API it was simple to have multiple copies and multiple versions running together in a cluster. But that has become more difficult with time, especially with ConversionWebhooks - a feature that forces all controllers for an API to agree not just on a version, but on the specific ways versions convert into each other.