[was_boring]

Thanks for sharing the second link. It's the first time seeing it an encapsulates my own philosophy on alerting well, while providing examples of what to alert on.

I've been with teams who treat alerts as informational and then quickly get alert fatigue. Then when something actually does happen they don't recognize it because they've trained themselves to ignore all alerts.

At my current company we regularly review our alerts to make sure they are actionable and meaningful still.

[yashap]

No problem! And agreed, the advice on “alert on symptoms users actually experience, and only on things a human actually has to deal with” is great advice. Alerts that don’t fit this should be quickly tweaked or disabled, to avoid alert fatigue.

“Something went wrong, but at acceptable levels/without much user impact” belongs in dashboards, not alerts. Worth being able to monitor, but alerts are for urgency, and that’s not urgent.