[db1024]

I wouldn't call it stupid. It was fine for when they wrote it. I would say their solution is a bit outdated though. I am a bit surprised that NVIDIA isn't allocating more engineering support to Caffe to make sure they support all of NVIDIA's latest features in a timely fashion. Torch at least tries to use nccl for nn.DataParallelTable by default.

[varelse]

IMO no it wasn't. Ring reductions are the sort of things the MPI guys had a decade ago. This is (once again IMO) the sort of thing that happens when one doesn't do lit searches and then Juergen Schmidhuber has a temper tantrum when we all find out one of his students probably has a good claim to have previously invented something supposedly new.

My heuristic whenever I think I've invented something is to aggressively exploit Google to prove me wrong. It usually does so, but sometimes in very unusual ways. Glad to hear about Torch though. Do they have automagic model parallelization (not data parallelization) as well?

[db1024]

> Do they have automagic model parallelization (not data parallelization) as well?

Not that I know of.

Regarding claims of novelty, I don't think the Caffe maintainers are claiming that their multi-gpu update method is novel or even very good. I think it was just the easiest thing someone could think of. I think Flickr originally wrote the multi-gpu extensions and the maintainers simply accepted the pull request.

If anything, I think the maintainers are more than willing to listen to people in the scientific computing community with experience. Even better if they have a pull request in hand. But otherwise, they probably won't know about better methods and won't care.

[varelse]

So from what I've heard unofficially from friends at NVIDIA, they've been pretty hard to work with such that NVIDIA just ended up making their own fork of Caffe for use within DIGITS.

https://github.com/NVIDIA/caffe

Am I missing something here?