[Veserv]

It uses FEE (function entry/exit) tracing so it generates a full execution trace of all functions called during your run like “History” [1], the C/C++ tracing profiler it was based on.

You can then post-process that trivially to get low resolution aggregate information like normal statistical/sampling profilers generate.

[1] https://www.ghs.com/products/MULTI_IDE.html

[kapilvt]

compare to pyspy, which is out of process and reads process memory to reconstruct stack frames as a sampler, which means marginal overhead for the program, re safer to use in prod. it would be interesting to see a function entry/exit on linux via epbf attached to USDT.

[Veserv]

They claim tracing overhead is <10%. This is very believable, and actually frankly seems kind of high. You can do a full execution trace of a C program for a similar low double digit percent overhead and Python is usually on the order of 10x slower, so naively you would assume like 1% overhead.

I assume the hooks Python makes available are probably just kind of bad for this use case and they do not have access to a proper high speed log.

[teh_matt]

Yeah, it tends to be low single digit percents, but you can get some pessimal behavior if you have enough tiny functions because the overhead of what Cpython exposes starts becoming large. Our "benchmark" where the 10% comes from is tensorflow, which is overwhelmingly tiny functions and no sustained io/individual functions.