Hacker News new | ask | show | jobs
by captainmuon 4671 days ago
You can use a custom memory allocator in python? I wonder if this is somehow pluggable, or if they had to modify the interpreter.
3 comments

All you need to do is set the appropriate type's tp_alloc/tp_dealloc function pointers [1] (type-specific malloc/free functions). Dropbox was having fragmentation issues from the large amount of garbage generated while scanning the filesystem, and making memory allocation use type-specific memory pools fixed it.

    [1] http://docs.python.org/2/c-api/typeobj.html#PyTypeObject.tp_alloc
The dropbox guys gave a talk about this at pycon one year but I'm having trouble finding it now. I remember thinking it involved less work than I thought it would.
According to the article, they do use a modified interpreter.