Currently we train on AWS EC2 instances. In the very beginning I was training on the GTX 980Ti in my desktop, which actually performed way better than I expected. Models trained faster on that machine than on p2.xlarge instances on EC2. But the advantage of training multiple models simultaneously, and using multi-gpu machines made the switch to EC2 worth it.