[shoo]

> "canary" means a set of basic automated integration tests that are continually running in production with alarms that feed into a master aggregate "switch". Wether the dedicated canary accounts end up hitting a one-box prod host or real prod host in the end isn't a factor.

this sounds roughly like synthetic monitoring:

https://en.wikipedia.org/wiki/Synthetic_monitoring

synthetic in the sense of synthetic traffic, since it isn't traffic from genuine users.

Can you go into more detail about what is meant by this:

> alarms that feed into a master aggregate "switch"

what is the master aggregate "switch" ? what does it do?

[beebs93]

> synthetic in the sense of synthetic traffic, since it isn't traffic from genuine users.

Yup - I think that lines up.

> what is the master aggregate "switch" ? what does it do?

We have a hierarchy of aggregrate monitors (or "switches") that watch n amount of either specific metrics or other sub-aggregate monitors.

In the case of production deployments, we watch a specific rollback aggregrate monitor for either a fixed amount of time or customer traffic that will auto-trigger a rollback if it goes into alarm (aka switches on).

We also have a master aggregrate monitor that will switch on if any sub-monitors get swtiched on for any reason. We typically watch this master aggregate alarm to auto-disable any promotions in our code pipeline.