[visarga]

> that still adds up to $2,451,526.58 to run 1,024 A100 GPUs for 34 days

Salary costs are probably even higher than compute costs. Automatic Speech Recognition is an industrial scale application, it costs a lot to train, but so do many other projects in different fields. How expensive is a plane or a ship? How much can a single building cost? A rocket launch?

[6gvONxR4sf7o]

In what way are salary costs higher? This is on the order of 10 of their people’s annual salaries. This is for a single training run (meaning overall compute costs are higher), and it isn’t the only thing those ten or so people would have done that year (also meaning overall compute costs are higher).

[visarga]

You have to work for a long time with a whole team on such a model. It adds up. In my experience there is a lot of work - data pipeline, labeling and data quality, the training, evaluations, deployment and making the deployed model efficient. And then there is also bias analysis, collecting failure cases and iterating on the data engine, and trivial things like measuring usage and billing.

[cpill]

yeah, but a cluster of the resulting model can transcribe thousands of hours of speech / second, 24/7 with a fixed accuracy, what can 10 humans do?

[6gvONxR4sf7o]

Huh. You and I have two very different readings on this. I'm talking about the ML researcher's time (what if they hired more people instead) and you're talking about human text processors (what if people did this work by hand instead). Kinda neat that we had such different readings.

[somebodythere]

This is an entire seed round's worth of money on an operational expenditure.

[dylanbfox]

> Salary costs are probably even higher than compute costs.

Yes exactly. Managing that much compute requires many humans!

[dinvlad]

I wouldn't be so sure :-)