[jrm4]

No, what I'm saying is -- it's kind of like Linux.

Yes, it would cost kabillions to recreate Linux from scratch. But no one has to anymore because it's here and anyone can just grab it.

Same with the models. What I can download right now from Huggingface is going to be 90% or more already there, and I can tweak them at home, despite how much it may have cost to make them in the first place.

[nativeit]

The balance may also shift towards more highly specific data that tunes local models towards the individual user. If that ultimately becomes more useful than, say, purchasing rights to some large archive of copyrighted materials, then things may lean farther towards a shared foundation upon which most day-to-day applications are developed. It will likely depend on the application and its specific focus, I don’t know that things will settle into an either/or situation, rather than a both/and.

[smokel]

I see. But what about adding new knowledge to the models?

To be useful in practice, one must train with new data, say every year or so. Who is going to pay for harvesting and cleaning all that data? And as I understand it, some form of RLHF is also required, which involves some manual labour as well.

Perhaps some day this will all be very cheap, and it might even be done by AI systems instead of humans, but I wouldn't bet my horse on it.

[jrm4]

I mean, given that I've literally done some version of it at home with RAG and it wasn't terrible, it's hard to think that it will be super difficult?

As in, I may not be able to do "general knowledge," but who will pay for that anyway (e.g. instead of baking it in to their Google killer?)