[merb]

Actually I never found Python "slow". I found it "slow" in some things, but not "all things are slow with python". What was problematic was pulling big lists from a database and doing some stuff with the list.

Also it was really akward that "threading" is not really great on python. Especially not when you are using 16 core servers. I mean you could create vm's for that or dockerize that. but that means deployment complexity increases which wasn't our goal. But I've seen a lot of successful python deployments and if you have enough manpower you pretty sure can run with CPython just fine.

[chrisseaton]

What you mean is that Python is fast enough for your purposes, which is great for you but it's not what this post is about.

I think it's pretty non-controversial to say that basic language operations in Python are slower than in other language implementations, which is what this post is talking about.

[brianwawok]

On a 16 core server you can run 32 copies of your program Ala Celery. It will peg the CPUs just dandy..

[retrogradeorbit]

My experience is that it is not "dandy". We have had a lot of trouble pegging the CPUs on our celery worker boxes (doing CPU bound jobs). You get more than one CPU utilised, sure, but we never can seem to get all cores fully utilised. We rewrote some of the tasks into a single multi-threaded JVM process pulling off the rabbit queue and they instantly and consistently pegged every CPU at 100%. I wish I knew how to get our celery worker farm to full utilisation because it would save us a fair bit of money.

[jstanley]

He's saying run 32 individual processes, not 32 threads within one process. Python's global interpreter lock will knobble you if you're using threads.

[darpa_escapee]

Celery has a worker pool of separate Python processes that jobs can be offloaded to. It side steps the GIL because it doesn't use threading.

[brianwawok]

Well two ways. Either launch 32 or 64 copies of the process using multiprocessing, or 1000 threads with geventlet.

I have never found a box I couldn't peg :)

I guess if you had 1 gb ram and 16 cores it would be challenging in python.. but a few gb of ram and we are on.

[retrogradeorbit]

We are using processes in celery, not threads. Yet we still can't get to full utilisation. On an eight CPU machine we get around 650% - 750% CPU utilisation. It varies, it goes up and down in that range. We never get to 800%. The JVM version is just constantly 800%.