Will it be possible to do LLM Pool Training?

[slicktux]

Inspired by Crypto Pool Mining: would it be possible to train LLMs as a pool of volunteers? I feel like that would be a good way to have frontier models that are open and don’t rely on big corporations with data centers to train and release them…

[bjourne]

Datacenter-scale GPGPU compute is extremely efficient compared to what you would achieve with a network of independently operated distributed heterogeneous devices spread across the world. Just imagine the overhead for ensuring that a node doesn't poison the gradients. And that's just one very small part of the problem.

[verdverm]

possible, yes, people already have projects

practical, not yet, if ever (?)

the difference is data and calculation dependencies, in crypto they are independent