| Maybe not $1 billion, but you'd want quite a few million. According to [1] a 70B model needs $1.7 million of GPU time. And when you spend that - you don't know if your model will be a damp squib like Bard's original release. Or if you've scraped the wrong stuff from the internet, and you'll get shitty results because you didn't train on a million pirated ebooks. Or if your competitors have a multimodal model, and you really ought to be training on images too. So you'd want to be ready to spend $1.7 million more than once. You'll also probably want $$$$ to pay a bunch of humans to choose between responses for human feedback to fine-tune the results. And you can't use the cheapest workers for that, if you need great english language skills and want them to evaluate long responses. And if you become successful, maybe you'll also want $$$$ for lawyers after you trained on all those pirated ebooks. And of course you'll need employees - the kind of employees who are very much in demand right now. You might not need billions, but $10M would be a shoestring budget. [1] https://twitter.com/moinnadeem/status/1681371166999707648 |
This just screams to me that we don’t have a clue what we’re doing. We know how to build various model architectures and train them, but if we can’t even roughly predict how they’ll perform then that really says a lot about our lack of understanding.
Most of the people replying to my original comment seem to have dropped the “in principle” qualifier when interpreting my remarks. That’s quite frustrating because it changes the whole meaning of my comment. I think the answer is that there isn’t anything in principle stopping us from cheaply training powerful AIs. We just don’t know how to do it at this point.