[bostik]

At this point Prometheus is pretty close to becoming the boring technology. The latest versions have finally brought in the plumbing and tuning knobs to protect against [most] overly expensive queries. So you can't easily take it down anymore.

The single-binary approach is still a problem, though. In my mind any serious telemetry collection stack should separate the query engine and ingestion path from each other - Prometheus has both the query interface and the ingestion/writing subsystem in the same process.[ß]

As for the parent poster: you certainly want to push telemetry out on every event, but the mechanism has to be VERY lightweight. With prometheus the solution is to have a telemetry collection/aggregation agent on the host, feed it with the event data and have prometheus scrape the agent. Statsd with the KV extension is a great protocol for shoveling the telemetry out from the process and into the agent.

ß: you can get around this with Thanos + Trickster to take care of the read path only, but it's quite a bit more complex than plain Prometheus.

[roskilli]

M3 separates query and ingestion if you're interested in clustered storage for metrics, slide in question here: https://www.slideshare.net/RobSkillington/fosdem-2019-m3-pro...

[bostik]

Oh, nice. Thank you. I had bookmarked M3 earlier on, but never got around to really looking into it with proper thought.

[NikolaeVarius]

I heavily disagree with Prometheus being boring tech. The storage backends still have heavy churn.

[wikibob]

See the Cortex project