|
|
|
|
|
|
by kernel_sanders
4844 days ago
|
|
|
This isn't uncommon in the python community for long running processes - especially those that frequently that create large amounts of small objects. An application server sitting in front of mongodb does exactly this. It forwards queries to mongo, then pymongo json loads the result for you, the application server does what it wants with the objects, then json dumps them to serve to clients. The json loads/dumps calls create tons of objects and memory bloat. The GC cleans them all up, yet memory isn't returned to the OS. See here: http://effbot.org/pyfaq/why-doesnt-python-release-the-memory... In my tests of python 2.6.6, 2.7.3, and 3.3, python 2.7.3 was by far the worst in hanging on to memory. Yet the performance increase (due to integration of simplejson) is worth the memory penalty. We use gunicorn with sync workers to serve our WSGI app and a memory watchdog to signal the worker to gracefully retire after it handles a query large enough to leave it with a large memory footprint. Performance has been awesome and we're very happy with the result. |
|
|