[nkallen]

Can you design the system such that you can turn off ("darkmode") features in an emergency? For example, in an emergency you can turn off badge-awarding as it is known to be RAM intensive (and degrade by seeking to disk).

To the extent that you do this across the board, you'll have yet another tool to defend against being over capacity.

[harryh]

We have a framework for doing this sort of thing but it's only implemented around certain features so it doesn't provide us a ton of benefit at the moment. Obviously doing work to give us more control like this could be a big win, so we'll probably be doing some of that.

[nkallen]

There is no better time than during an outage to add these features. In my experience, band-aids are better way to resolve an outage than fixing root-causes. Make the problem go away by commenting out code, returning empty result-sets, (or whatever) - and re-balance your partitioned database at some later date.