[sigil]

...we are able to perform all our maintenance without downtime. We can deploy new versions of our software, make database schema changes, or even rotate our primary database server, all without failing to respond to a single request. We can accomplish this because we gave ourselves the ability suspend our traffic. To make this happen, we built a custom HTTP server and application dispatching infrastructure around Python's Tornado and Redis.

Why is it necessary to suspend traffic to make these kinds of changes? Just curious.

[ary]

Probably because the data transformations and storage required to complete a transaction need to be handled by a coherent version of their code. Processing payments with a half-updated stack sounds painful and error-prone.

[fleitz]

Because schema changes and deployments are generally non-atomic, running a transaction through an untested configuration (front-end code = 1.0, schema = 1.2 and backend = 1.0) generally results in unknown things happening. When dealing with money people generally prefer right to fast.

[Swannie]

This one made me go "Huh?" and reread.

I don't understand why you'd need a custom HTTP server. It might be overly simplistic, but just write a module for nginx? Surely that would have been faster. You get controllable timeouts, you could tweak their throttling code to stop it throwing 50x, and modify/take inspiration from nginx-ey-balancer. And you get a bullet proof HTTP engine for free?