[tiger10guy]

For the every day when I want to make embarrassingly parallel operations in Python go fast I find joblib to be a pretty good solution. It doesn't work for everything, but it's quick and simple where it does work.

https://pythonhosted.org/joblib/

[shogunmike]

I was going to discuss Parallel Python (http://www.parallelpython.com/) in the next article - have you used that? How does it compare to joblib?

[tiger10guy]

I haven't used that, but it looks interesting. After a brief look it seems like they both submit jobs to Python interpreters started up in other processes.

Parallel Python (PP) seems to have a clunkier API, but also more functionality. I think the biggest advantage is that it can distribute jobs over a cluster instead of just different cores on the same machine. I might look into PP if I need to do things on a cluster, but I think I'll still stick with joblib when I'm on one machine.

That's just my first impression. I'd be interested to read your blog post.

[harlowja]

Feel free to checkout http://docs.openstack.org/developer/taskflow/ (it has a similar set of concepts to both of the mentioned libraries here, joblib and parallel python).