[agibsonccc]

Keep in mind there's different kinds of parallelism though. If you mean model parallel, a lot of shops are doing that via RDMA as well as MPI. It depends on how you handle state though.

There's also data parallelism with parameter averaging which we've been doing in deeplearning4j for the last few years. We also support ALOT more than just images. We have the ETL pipelines (kafka etc) to go with it. Watch for a blog post from us on parallel for all (nvidia's blog) where we explain some of this.

I gave a framework agnostic view of the concepts you should consider when looking at distributed deep learning as well:

http://www.slideshare.net/agibsonccc/brief-introduction-to-d...