[jamesblonde]

Great talk, with lots of new insights into what's happening at Google. I really think his point that ImageNet is the new Mnist now holds true. Even research labs should be buying DeepLearning11 servers (10 x 1080Ti) for $15k, and training large models in a reasonable amount of time. It may seem that Google are way ahead, but they are just doing synchronous SGD, and it was interesting to see the drop in prediction accuracy from 128 TPU2 cores to 256 TPU2 cores for ImageNet (76 -> 75% accuracy). So, the algorithms for dist. training aren't unknown, and with cheap hardware like the DL11 server, many well-financed research groups can compete with this.

[eggie5]

ballpark how much would it cost to train ImageNet (ILSVRC) on a std deep CNN arch (VGG or inception) on AWS using a p2 or p3?

[jamesblonde]

Ballpark - 1100 dollars on AWS. 44hr 28min (from Dawnbench - http://dawn.cs.stanford.edu/benchmark/ ) on a DGX-1 (cost 24.48 dollars/hour on p3.16xlarge). https://aws.amazon.com/ec2/pricing/on-demand/

On a DL11 server, it will take about 60 hrs, and only cost you 15k upfront. The economics speak for themselves for fp32 training, at this moment in time.

[eggie5]

I didn't know about the dawn project, thank you for the reference and figures.