[Gladdyu]

I'd say it's a problem in the interaction of that specific version of python with that specific malloc. From a performance perspective we could not afford to simply force malloc to only use mmap for every small allocation (as it would thrash the TLB and/or balloon memory usage).

Something they could have done inside Python is manually implement a short-lived objects space that lives entirely in pre-allocated memory, preventing the repeated allocations.

I should have mentioned that the app in particular was running python2.7 - some preliminary tests on porting to python3 seemed to indicate that the issue was significantly less severe there, though the problem only occurred very infrequently that restarting the affected processes when they started exceeding a memory threshold was a more cost-effective solution that spending ages debugging the interactions.

[masklinn]

> I'd say it's a problem in the interaction of that specific version of python with that specific malloc. From a performance perspective we could not afford to simply force malloc to only use mmap for every small allocation (as it would thrash the TLB and/or balloon memory usage).

And other option might have been to switch to a malloc using mmap'ed size-class arenas like jemalloc or somesuch?