Hacker News new | ask | show | jobs
by beebs93 2421 days ago
Yeah, very useful strategy. I've heard it go by various names (e.g. toe-dipping, one-boxing, etc.), but it's always been one of additional methods of helping ensuring safe prod deployments at any company I've worked for.

One downside - depending on your setup - is you may not have an easy way to hit the hosts directly/deterministically via any UIs in case you wanted to do any manual verification/debugging yourself.
1 comments
sagichmal 2421 days ago
> Yeah, very useful strategy. I've heard it go by various names (e.g. toe-dipping, one-boxing, etc.)

Strange terms. Isn't this just a canary?

link
beebs93 2421 days ago
Every company I've been with seems to re-invent the terms or swich their definitions slightly.

For me, currently, "canary" means a set of basic automated integration tests that are continually running in production with alarms that feed into a master aggregate "switch". Wether the dedicated canary accounts end up hitting a one-box prod host or real prod host in the end isn't a factor.

The important thing is we incrementally expose our latest code commit to prod hosts via one-boxing to reduce the customer exposure if an acute problem somehow gets past the previous code deploy stages/tests.

link
shoo 2420 days ago
> "canary" means a set of basic automated integration tests that are continually running in production with alarms that feed into a master aggregate "switch". Wether the dedicated canary accounts end up hitting a one-box prod host or real prod host in the end isn't a factor.

this sounds roughly like synthetic monitoring:

https://en.wikipedia.org/wiki/Synthetic_monitoring

synthetic in the sense of synthetic traffic, since it isn't traffic from genuine users.

Can you go into more detail about what is meant by this:

> alarms that feed into a master aggregate "switch"

what is the master aggregate "switch" ? what does it do?

link
beebs93 2420 days ago
> synthetic in the sense of synthetic traffic, since it isn't traffic from genuine users.

Yup - I think that lines up.

> what is the master aggregate "switch" ? what does it do?

We have a hierarchy of aggregrate monitors (or "switches") that watch n amount of either specific metrics or other sub-aggregate monitors.

In the case of production deployments, we watch a specific rollback aggregrate monitor for either a fixed amount of time or customer traffic that will auto-trigger a rollback if it goes into alarm (aka switches on).

We also have a master aggregrate monitor that will switch on if any sub-monitors get swtiched on for any reason. We typically watch this master aggregate alarm to auto-disable any promotions in our code pipeline.

link
sagichmal 2420 days ago
> For me, currently, "canary" means a set of basic automated integration tests that are continually running in production with alarms that feed into a master aggregate "switch".

I've never heard of this practice described as a canary before (shrug)

link
beebs93 2420 days ago
Yeah, I concede the common lingo within the company (or team even) may not align with either proper definitions or mainstream usage.
link