Hacker News new | ask | show | jobs
by INTPenis 1545 days ago
What issues have you seen with Grafana alerting?

I'm curious because in my view it works so well that we abandoned alertmanager for Grafana alerts only well before v8.

3 comments

Grafana alerts (before version 8) worked great. We use them, but the Grafana 8 alerting features are half-baked at best.

* Grafana 8 alerts removed the Image Preview, which was extremely useful during issues. [0]

* Grafana 8 alerts don't have any way of being stored as code. In fact the API that they provide in their docs [0][1] doesn't work, or isn't up to date.

* The expression languages have zero documentation about them, so aren't exactly useful for things that might get a developer out of bed in the middle of the night.

[0] https://github.com/grafana/grafana/discussions/38030#discuss...

[1] https://editor.swagger.io/?url=https://raw.githubusercontent...

[2] https://community.grafana.com/t/posting-an-alert-using-grafa...

How did you define alarms as code in a practical way before v8? and after?
Hi, I work on Grafana Alerting. Provisioning of alert rules (and other objects used for alerting) will be possible using a new API in Grafana 8.5 and we will update the Grafana Terraform provider right after to take advantage of this new API.
Great to hear! We are looking into jsonnet based approach but having an explicit and granular API and a Terraform provider would be miles and miles better. Thanks!
Did not tbh. We have an ops department that do not complain about menial tasks.

But of course IaC is the way we must follow.

Building a dashboard by clickety/clacking around is not a menial task, consistency across dashboards is a a core unit of observability to ensure x-functional teams can discuss issues across a common language/viewpoint, which is only enforceable through a declarative dashboard syntax.
The question was regarding alerts, not dashboards. We obviously deploy dashboards from json.

But I'm not aware of any way to deploy notification channels, probably can do that now via API. But either way we need to deploy notification channels with webhooks and tokens so that part is done manually. And then the alerts is also done manually.

No alerts possible with dashboards and variables.