[kragen]

This is very insightful. I remember the epoch of clueless startups wasting venture capital on Sun servers. I worked at one of those startups. Warden is clearly correct that if you want to train your AI faster then the optimal amount to spend on software optimization is at least a substantial fraction of your hardware budget.

However, clueless people who don't know how to optimize probably don't know where to spend money on optimization, either. So maybe it's just not a great fit for outsourcing, especially in a realm where there's no standard of correctness to measure the results of the supposedly "optimized" training against. And Warden seems to be pitching outsourcing rather than trying to get acquihired.

[JohnBooty]

    the optimal amount to spend on software optimization 
    is at least a substantial fraction of your hardware budget.

This has been a banging-head-against-wall sort of struggle every place I've worked on software, without AI even coming into the picture.

At one startup they were spending millions of dollars on AWS and complaining loudly to us about AWS spend and yet... god forbid the engineering team devote any resources to optimization passes instead of rolling out more poorly considered features, and hiring more engineers, because the existing engineers are struggling to be productive because everything is so unoptimized, and also because they have to spend a bunch of their time interviewing and training new hires.