[dpkirchner]

Neither of those cases really need an operator -- Prometheus and cert-manager both have code that watches for changes on ingresses/services/custom resources and reacts to changes (using permissions granted via RBAC). I've used both without an operator and still use Prometheus without one.

[ecnahc515]

Everyone else commenting is missing the point, because as you've said, Prometheus can discover targets via k8s endpoints/pods dynamically already, and you can run a sidecar to reload the config file.

The main point of Prometheus operator is to federate access to Prometheus configuration so teams can manage configuration that the built-in Kubernetes service discovery doesn't let you control from an application side. Things like scrape interval, recording rules and alerts, etc. ServiceMonitors, PrometheusRule CRDs basically let app owners ship these aspects of monitoring with their applications, instead of having to have a hand-off between the service owners and the SRE teams managing the monitoring infrastructure.

[pojzon]

Pronetheus does not change its config if not asked to do it… and in most cases scrape config is stored in a configmap that does not change by itself magically.

So yes -> you need an operator to dynamically change config of prometheus. If not -> all your changes wont matter because they will be not reflected in prom.

[rustyminnow]

There are several Prometheus helm charts out there. The one I use runs a sidecar container that watches for config changes and tells Prometheus to reload the config. So you don't need the operator - there are other approaches.

[pojzon]

But that configmap does not change by itself.

The same with annotations approach - its very limited.

[rustyminnow]

Wdym the configmap doesn't change by itself? I don't want my configurations changing "magically" without my say so

The two features I need (and can get without the provider) are: 1. If I (manually, explicitly) change the configmap, prometheus can pick it up without being restarted. This is provided by the side-car container. 2. If new pods come online they are automatically detected and scraped by Prometheus. As long as they have the annotations saying: a) this pod should be scraped, b) scrape on this port, and c) scrape at this URL - then they will be scraped.

[dpkirchner]

I dunno what to tell ya, I see pods come and go from Prometheus all the time, including those from newly added Service objects with Prometheus annotations. I'm using kubernetes_sd_configs.

[rustyminnow]

You don't NEED an operator, but there are benefits.

If all of your pods have the same scrape settings, there's not much benefit to the operator. But if each pod or set of pods needs custom scrape settings - say different scrape intervals or custom tag re-writes, the operator will let you define each set of scrape configs separately as kubernetes resources. You can store the definitions alongside the resources they scrape instead of having a single large complex config in your prometheus deployement. This would be especially beneficial if you have multiple projects owned by multiple teams, all scraped by the same Prometheus.

Granted I also don't use the operator, but I've looked into it. With complex enough deployments it would simplify things.