[jrichardshaw]

I'm a big fan of pyinstrument. Many of the newer profilers, (e.g. py-spy) attach to a process externally via SYS_PTRACE and though that seems great in many ways, it is very much a no-go when you're running code on an HPC cluster and you don't have root access.

[gnufx]

On an HPC system, why wouldn't you use normal HPC-type performance tools? It will probably give you better analysis tools and allow you to profile/trace the libraries which are probably providing the performance. At least Score-p, Extrae, and TAU have Python support and have sophisticated viewers with support for call trees and inclusive/exclusive views. When I looked (maybe a couple of years ago) I couldn't find anything in the Python ecosystem to do that, though there may be now.

[kapilvt]

ptrace is quite slow, py-spy reads memory and reconstructs stack frames which is much lower overhead for the profiled app.

from their github readme:

  py-spy works by directly reading the memory of the python program using the process_vm_readv system call on Linux, the vm_read call on OSX or the ReadProcessMemory call on Windows.