[darren0]

I would love to understand the design rational on why a custom controller is needed to run etcd as opposed to leveraging the existing k8s constructs such as replicasets or petsets. While this is a very useful piece of technology it gives me the wrong impression that if you want to run a persistent or "more complicated" work load then you must develop a significant amount of code for that to work on k8s. Which I don't believe is the case, which I why I'm asking why this route was chosen.

[theptip]

The FAQ at the end of the OP addresses this:

"Q: How is this different than StatefulSets (previously PetSets)?

A: StatefulSets are designed to enable support in Kubernetes for applications that require the cluster to give them "stateful resources" like static IPs and storage. Applications that need this more stateful deployment model still need Operator automation to alert and act on failure, backup, or reconfigure. So, an Operator for applications needing these deployment properties and could use StatefulSets instead of leveraging ReplicaSets or Deployments."

There is inevitably some app-specific logic required to modify a complex stateful deployment; the Operator encapsulates this logic so that the external interface is a simple config file.

[philips]

Darren, the FAQ is at the overview post here: https://coreos.com/blog/introducing-operators.html

[darren0]

Thanks, but this alludes to more operators coming. Why is such a thing really needed. The proliferation of this approach seems like a potential downfall of k8s. Similar to Mesos where a framework is quite powerful, the cost of developing one is too high. This blog post basically implies that k8s base constructs can't run postgres, redis, prometheus, etcd, cassandra, etc. But why? Are we saying that stateful services fundamentally require one off domain specific logic to run in k8s?

[philips]

I don't think stateful applications require the use of something like an Operator. It really just comes down to where the state lives. For example if you are wanting to run your database on top of an EBS, SAN, or something like that it is no problems to just throw up a StatefulSet and go for it.

However, if you start to think about things like orchestrating scaling of databases that have a administrative tool like Cassandra, Vitess, RethinkDB, read-replicated Postgres, etc you need to have some glue/gel that integrates those admin tools with the Kubernetes APIs. And that is what a minimal Operator should do.

The other thing Operators can do is glue existing software to the Kubernetes APIs. Which is what the Prometheus Operator does. Prometheus has its own configuration system for finding monitoring targets; instead of forcing the user to drop down into that different format the Operator adapts Kubernetes concepts like label queries and generates the equivalent Prometheus config.

Overall I don't think this is required for every application. Static databases persisting to shared storage, stateless applications, or cluster wide daemons all fit nicely in Kubernetes abstractions before Operators. But, there is a class of clustered applications that are served well by this pattern.

[tristanz]

I think philips is exactly right. The whole design of Kubernetes is geared toward allowing users write their own controllers for advanced use cases. You could view something like Jenkins or Vitess as controllers because they spawn Kubernetes pods on demand. The beauty is Kubernetes gives you great primitives, so you often will be controlling these objects, not the underlying pods. Of course many simple applications don't need a controller, although I suspect more and more simple use cases will be managed by an external controller like Helm which orchestrates the lifecycle of applications.

The questions is: Why isn't this just called a controller? What's this new term Operator?

[philips]

Controller wasn't quite the term to capture the combination of an application specific controller and third party resource to manage a collection of user created application instances.

So, we arrived at Operator. It felt like a good term that we could put after X that encapsulates the intent of the pattern. It helps you operate instances of an application.

[invisible]

The reason is that while replication controllers and daemonsets fulfill making sure there are enough of xyz pod, they do not ensure that replication is set up between those pods. For example, in redis one must do `slaveof` for slaves if implementing basic replication. If the master goes down, a new master must be appointed and then `slaveof` configured on afterward. That in itself requires either a) an operator or b) concensus among enough nodes to reach a quorom about these.

So while kubernetes could be the authority (via the master), generally it's only meant for scheduling pods. There would have to be some better guarantees about master availability if it were to control these things.

[Terretta]

"An Operator represents human operational knowledge in software, to reliably manage an application."

Loving this.

[SEJeff]

Do you ever see this pattern being used to host/run the etcd cluster for the k8s apiserver to speak with? That would be quite meta (and kind of amazing if it would actually work).

[philips]

This is in progress! We have it prototyped out in bootkube. This is the plan. :)

[SEJeff]

Oh this is absolutely brilliant! So with the kubelet + systemd, you more or less have self managed everything. Shutup and take my money :)