[spamizbad]

We switched from Datadog to Grafana (do not recommend unless they got you over a barrel on pricing and you need to escape) and one nice thing Grafana gives you is the ability to self-host for local development so you can even run integration tests against your observability... an edge case need but if you need it you're glad it has it.

[samjewell]

I work at Grafana. Can you say more about what specifically you don’t recommend?

[benreesman]

I’ve historically been a pretty big fan of Grafana, I’ve advocated for the cloud solution at more than one company.

But it seems like business development has utterly hijacked the experience.

The flow you want out of the box is Prometheus, Loki, and Tempo with one button that drops you the config for grafana-agent (now alloy which seems good technically but brings a whole new config language with some truly insane discoverability problems) that makes graphs on screens, you build up from there.

But these days everything is some complicated co-sell, up-sell, click farming hedge maze through 90 kinds of cloud vendor rip-off half baked thing.

Graphs, logs, traces out of the box. Put all the works with Snowflake shit behind an icon. A small one.

[Multicomp]

Not OP, but looked at doing grafana self hosted for similar reasons. The tooling is too spread out across different installables, the common golden signals and other monitoring metrics have a high learning curve / cliff more like it, and there's not good enough documentation to cover the user from "I want to do a synthetics test on a service to see if it is alive and show the results of that test in a graph"...a journey like that involves 4+ different tools.

[spamizbad]

DD was just easier to use for everybody, has lots of useful baked-in things we liked to use (apdex scores), and was intuitive enough that non-devs could design their own dashboards. We also found it way easier to collect metrics, traces, and other things.

FWIW none of these things are insurmountable and I suspect you'll eventually reach parity. Datadog lost our business for two reasons 1) Lack of billing transparency and 2) an incompetent account rep who managed to piss off our finance department while also embarrassing our CTO. And to be clear, while we aren't a "whale" our spend was over $6M/yr with Datadog - and the CTO along with the rest of eng leadership were all huge DD fanboys and yet they still managed to burn that bridge to the point where we'll never go back.