[NovemberWhiskey]

We have a grid compute infrastructure for a specialized runtime environment with business-logic rules for scheduling priorities and partitioning of the compute cluster.

The control plane was implemented in Python and Twisted (event driven I/O framework for the unfamiliar), which was fit for purpose at the original scale running CPython (few thousand compute nodes).

As the number of compute nodes scaled up, we developed hotspots in ser/des of control messages, which ultimately started to affect overall cluster efficiency.

Switching to PyPy gave us an immediate substantial performance boost without really having to redo any code at all (just some FFI stuff that was probably wrongly implemented in the first place).

Eventually we realized we were going to out-scale even that (at the hundreds-of-thousands of compute node level) and ended up with a Scala/Akka reimplementation, but moving to PyPy from CPython got us a lot of free breathing room.