[andoando]

Why isnt LLM training itself open sourced? With all the compute in the world, something like Folding@home here would be killer

[DesaiAshu]

data bandwidth limits distributed training under current architectures. really interesting implications if we can make progress on that

[dogcomplex]

Limits but doesn't prohibit. See https://www.primeintellect.ai/blog/intellect-3 - still useful and can scale enormously. Takes a particular shape and relies heavily on RL, but still big.

[andoando]

What bandwith limits? Im assuming the forward and backward passes have to be done sequentially?

[DesaiAshu]

Yes also passing data within each layer

[mike_hearn]

It is in some cases. NVIDIA's models are open source, in the truest sense that you can download the training set and training scripts and make your own.

[throwaway27448]

It's either illegal or extremely expensive to source quality training material.

[m4rtink]

Yeah, turns out if you want to train a model without scrapping and overloading the whole of Internet while ignoring all the licenses and basic decency is actually hard & expensive!

[doctorwho42]

Well it is, it's in the name "OpenAI". /S