It doesn't just need to be a place to experiment cheaply. Many companies building software around ML techniques still rent time on EC2. Unless you are training models 24/7 and have your machines located in a very cost efficient location in terms of power/cooling, It's probably better for your training to be done in the cloud. It think very few use cases fall into the latter category.