[notemaker]

I've written a couple of C-extensions for a Python service that needed to run in real-time. We started out 100x slower than real-time and managed to make it slightly faster than real-time in a hectic couple of weeks.

The main bottleneck that C-extensions solved for us then was marshaling. Instead of doing something like Numpy --> custom algorithm --> Numpy --> custom algorithm in Python, we moved it all to C. This way, we decreased the number of times we had to marshal data & we reduced the amount of data that needed to be marshaled.

IIRC we gained 10-100x improved performance in all scenarios where we used C-extensions. Before doing so we tried to use Numba, as it seemed to offer a "free lunch", but we never got it to work.

The final cherry on top that let us surpass real-time speed was replacing a graph algorithm with another that had a better time complexity for larger graphs. Before doing that switch, we'd still be faster than real-time for _most_ inputs but would lag behind severely whenever we had a dense input graph come in.

[packetlost]

It's not really meaningful to say "100x slower than real-time" or "slightly faster than real-time", "real-time" isn't some threshold of sub-second precision. It usually refers to relative (though maybe absolute wall-clock) time of execution for instructions that are typically controlling hardware that needs precise timings (ie. the time between turning this thing on and turning it back off needs to be exactly 150us).

[notemaker]

It is meaningful, but maybe I didn't give enough context.

The service ingests data from a timespan T spanning M minutes. Before the next timespan T+1 is ingested, the service has to process the data from timespan T in <M minutes otherwise we fall behind. What I mean with "faster than real-time" is that it has to process the data faster than it is ingested.