[dozzie]

> Should all environments be monitored?

Yes. If you don't collect data about the environment, you can't tell whether the current behaviour is normal or abnormal. Though not every environment should send all alerts possible.

> I was wondering how others handle these problems internally or if we are the only ones who are having this discussion?

Monitoring is a tough topic -- still, after more than twenty years of development in this field. The problem is that we still treat monitoring systems as monolithic applications with hardcoded data collection and alerting logic, instead of making them what they are: event stream processing engines, with some publish-subscribe capabilities to allow attaching some external logic.

I'm working on a toolbox that helps in building such a system (http://seismometer.net/), though it's still lacks tools and its website is more of a sketch than a project homepage.

In the meantime, you should think about how to allow different people to select or filter out different notifications, possibly in such a manner that they can process those notifications programmatically.