Hacker News new | ask | show | jobs
by mhils 2802 days ago
Instead of throttling, you could also consider doing incremental brownouts where you drop requests for the first ten minutes every hour. PyPI did this recently when they phased out TLS 1.0, which worked really well IMO.
3 comments

I wish more services shuttered this way because you can’t miss it.

I once had a web host generously give me three months to pay a delinquent bill that I missed the emails for. Sadly it just meant I thought things were fine. When they finally shut my service down, my users made me aware within minutes but it was too late.

Thanks for the suggestion! This does seem potentially more effective.
IIRC Google also uses (or used) a similar system for deprecating old APIs.

Start with failing 1% of requests randomly and slowly ramp up from there.

Incremental brownouts only work well if there is mechanisms to ensure that your service's users realize that the brownout is a deprecation warning.

By default, pip doesn't show the contents of HTTP error messages [1], so users affected by the brownout would have to take extra steps (using `-v`, visiting the PyPI status page) in order to figure out what was wrong. I think it could easily appear as a networking issue or some other sort of intermittent problem.

There was also no notification of the impending blackout on python.org. [2]

[1]: https://github.com/pypa/packaging-problems/issues/130

[2]: https://lwn.net/Articles/751800/