[bluegatty]

Making an LLM from raw data is value-add.

Distillation is just value extract.

It's soft, and I'm not sure what the answer should be ... but I think that there is a difference.

I think we start by recognizing that ... and then try to figure it out from there.

'The Internet' may be a public good, maybe we make them pay a tax for that, but that's different than distillation.

[scotty79]

> Making an LLM from raw data is value-add. > Distillation is just value extract.

There is a value-add in selecting the valuable parts out of the garbage. And let's face it. Largest models contain a lot of garbage.

[bluegatty]

I think that's kind of fair, but it still fits within the context of 'some things are value add' and 'more or less than others'.

We ought to identify that and integrate that into our thinking.

[nemomarx]

What makes the Internet raw data in a different way? wasn't it mostly worked on by people first?

[bluegatty]

There is value add in AI irrespective of how the data got to what it is.

Literally the biggest thing of our generation - AI - is the living embodiment of that 'value add' writ large.

'What is the difference' - is the AI you use all day, in comparison to 'all the world's data' you can use for stuff and do 'whatever' with it, but are not likely to come up with something hugely useful otherwise. Maybe, not likely, if you did, it would be 'value add'.

[nemomarx]

Okay, so if the chinese models are used everyday, do they become a value add? Like what's the line you're drawing here. Amount of value it creates?

[bluegatty]

Designing and creating an LLM from nothing is a monumental feat of Engineering and 'value add'.

Copying something is not.

Programming Microsoft Word is value add, copying the code is not.

Copying design ... there are some question marks there.

It's extremely easy to understand at it's core.

What makes it hard, is that faux intellectuals like to deconstruct ideas at the margins, and have those critiques stand in for reason.

"At sunrise the sun is only 'half there' ... there fore there is no 'day and night' just a blur! Day and night are the same thing!"

The training data used is part of all of this is a separate but related question.

[inigyou]

they all just copied the Transformers paper anyway

[bluegatty]

If all it took was 'copying the transformers paper' then why aren't there 1000 SOTA foundries?