[markbnj]

I'm wary of the operator model in general, and we haven't had great success using operators to deploy complex stateful services in our clusters. But to be honest we also haven't had great success deploying them using OTS charts from helm stable either. One of our k8s stateful services is a large elasticsearch cluster indexing about 150m events per day, and the chart was forked and heavily modified by us to get it right. I feel that complex stateful services often have enough devils in the details that trying to implement them through an abstraction gets you into trouble. Operators aspire to be a "smart agent" that can translate a CRD resource declaration into a functioning thing, allowing you to implement your data store at an even higher level of abstraction than a helm chart provides. Since in my experience charts are themselves too abstract for this purpose (you either end up forking/modifying or, if the chart actually provides full coverage of the configuration options, creating a whole new hard to comprehend API to the k8s resources you're trying to deploy), I'm not that excited about having a back-end clippie that can do it for us. It's probably fine for simple use cases, and especially those where you often need to create and destroy simple dbs, but imo not yet for large production use cases.

[marcc]

The Operator/CRD pattern is promising for autonomously operating simple use cases of existing software and for operating really complex software that needs very specific, rare knowledge to operate.

Unfortunately, we aren’t there yet for most software. Let’s take Postgres as an example. Even though you have to manage your pg database manually (or use a service that manages it for you), that’s just because the right automation software hasn’t been built yet. Someday, a Kubernetes Operator (or equivalent implementation) will exist that can manage a large Postgres cluster better than a team of DBAs. It’s crazy that there are hundreds (thousands?) of configuration parameters in Postgres, and these are coupled to the operating system settings in weird and unexpected ways that most people don’t know. We should be building this knowledge into a K8s Operator and letting that control our pg.conf and os configuration, instead of giving that control over to a team of humans who might be able to put in some sane defaults, but will always be working to get the optimal performance out of Postgres as the usage share changes.

This exists in some places already. For example, Rook is a K8s operator that provisions and manages Ceph in a Kubernetes cluster. As a small startup, if I need this functionality, I don't want to hire a full time Ceph admin to figure it out, and I don’t have the expertise to take on operating Ceph myself. Rook productized operating Ceph for us, and “baked in” all of the needed knowledge to manage block and object store and even set up concurrent, shared file systems. I trust Rook to manage Ceph, and I don’t think that I could do a better job with human intervention.

We have a long way to go. Operators are a tool that might help get us there but Operators are just a pattern that exists that we can use. One thing for sure is that we shouldn’t assume that human control over complex software is required to achieve optimal performance.

[andrenth]

What's your strategy for handling possible filesystem failure/corruption scenarios without a team that understands the underlying technology?

[marcc]

That's a great point. I do have a team that understands the underlying technologies and has been successful in troubleshooting several production problems with Rook/Ceph, one recent one including file system corruption. My original post is just trying to state that our engineering team does not maintain a deep operational knowledge of the best way to configure, manage, monitor, scale, etc (operate) ceph in production. We rely on the Rook operator for this.

Troubleshooting acute outages caused by hardware or software failures requires a different skill than properly configuring the system to scale and minimize the chances of a corruption or outages. Rook solves the later, but we do understand the architecture and what Rook (and Ceph) are doing. We've just removed the expert level, craftsman, speciality knowledge required to operator Ceph because we decided, after a thorough evaluation, that the software in this case is the most capable solution.

[andrenth]

I find this unusual because usually the knowledge require to troubleshoot a complex piece of software is much more complex than that required to set it up in the first place. In other words, how can you troubleshoot it if you don’t know how it’s built?

It’s a bit like debugging software you didn’t write.

[ec109685]

I agree you do not want to be running half baked operators on your cluster, but clouds like Amazon’s and Google’s have shown it is possible to harness open source software like ElasticSearch and successfully run it on behalf of customers.

It will take time for these to harden, but it eventually will since the primitives are all there.

That said, a very significant issue that the public clouds don’t face is the team creating the operator in aws / gcp’s case is the one running it and they fully understand exactly the configuration the operator will be deployed to.

With helm and operators designed for public consumption by other companies, the amount of generality needed will be higher, at least until the conformance tests for a kubernetes implantation get more detailed and there is ci/cd between the operators and charts running on clusters that better mirror various Kubernetes deployment choices.

[cyphar]

The thing that's always bothered me about the operator model is that you have to reinvent all of the things that Kubernetes does for its own objects (failure handling, redundancy, declarative configuration). At least that's my understanding, from what I've seen.

[smarterclayton]

Arguably all kube is just a pattern library, and the patterns fit reasonably well together. But patterns are situational - you’re always going to have to fill the 20% oc uncovered space with something unique to your use case. Operators are more like frameworks using those patterns. They can reduce time to value, but once you hit the areas where you are fighting the framework you might want to implement those patterns yourself.

When we designed stateful sets, it was to make the minimum bar easier to get unique network identity (which is necessary, but not sufficient for most cluster software). And in practice I’ve seen people using sets directly with a thin layer of scripting or helm on top, but I’ve also seen people implementing their own stateful sets because the logic isn’t that hard once pods had the necessary shims.

I would probably say that kube is best thought of as an extensible compute pattern framework (you can leverage the lowest level atoms or build layers on top). Kube is probably only successful as long as we keep the 80% easy, reduce the cost to add new patterns (libraries and tools), and decompose the bits that should be replaceable.

Most of the “reinventing” problems with operators are problems with kube having weak libraries - each controller and operator is somewhat bespoke. That’s something I expect to see improved this year via the various tooling libraries. But it’s still a work in progress and I regret it took so long.