The difference I was referring to was economic; I was not making an ethical judgment. So many people seem to be reading my comment as making an ethical judgment (based on their reactions); maybe I should edit it to clarify.
> The difference I was referring to was economic; I was not making an ethical judgment
What is the economic difference? LLMs have been trained on the results of billions of dollars worth of time, research, investment and expenditure. When you ask an LLM a question, they are giving you the results of those billions, or hundreds of billions, effort.
Those things weren't free; they cost money to produce! If anything, the OpenAI and Anthropics of the world got more economic value for free than the people distilling them did.
It takes more computing power, and money, to pay for training an LLM, compared to distilling an LLM that someone else has already trained. That's the economic difference.
Which is true, but also completely irrelevant to the point of this discussion. Which was about the U.S. companies spending trillions and the Chinese companies spending billions and matching them in quality. I made the point that the U.S. companies and Chinese companies were doing different things: training on raw data vs. distilling the model that someone else had trained. That difference explains the difference in spending. (The secondary point is that the Chinese companies would not have been able to get to the point they have, while spending as little as they have, without Claude, GPT, et al to copy from).
This is probably the last reply I'll make to you. I'm getting a little tired of repeating myself. Whether you're just not getting it, or refusing to get it, either way it's starting to feel like a waste of time to try to rephrase the same thing again and again. Please read more carefully in the future.
And, as I pointed out elsewhere, the difference I was pointing out was an economic difference. You keep on ignoring that fact and thinking I'm talking about ethics, but I'm not. I'm saying the difference is between spending trillions on training from raw data vs. spending billions on distilling that trained-from-raw-data model.
First of all, this is actually how this part of the conversation started:
> If the US companies need trillions to barely beat Chinese companies spending billions, despite a multi year head start...
Because they're talking about the cost difference of distilled model development and ground-up trained model development.
And second, the answer is that OpenAI and Anthropic had to do all the research into how to train models. Then they had to acquire all the data, curate and filter it. Then they had to design all the ways to iterate on training and antagonize it to be better - because there's not actually an enormous corpus of aligned, human stream-of-thought data. Then over half a decade they've been refining these methods.
There's no way you don't understand that if it was not easier and cheaper to distill a model, the institutions in question would be training their own models from scratch.
As opposed to all the copying AI companies have done of everyone else's work, in order to build something that tries to replace and undercuts the people who did that work?
Not saying the American AI companies are ethically in the right, just that the work they did is a lot more expensive to do than what the companies distilling their work are doing.
That's an ironic comment to make about LLMs that are heavily reliant on copying/distilling everyone's work (or as Damien Walter puts it, our Collective Intelligence).
Good UIs look alike, yes. But I'm not sure that comment is a good rebuttal in this particular case, because the model calling itself Claude is pretty strong evidence of distilling: https://x.com/denisewu/status/2077984660211269870
(I am Tongyi Qianwen (Qwen), a large language model independently developed by Tongyi Lab of Alibaba Group. I can help you answer questions, create text (such as writing stories, official documents, emails, scripts, etc.), perform logical reasoning, program, translate, and more.Is there anything I can help you with? )
Huh? What exactly do you think I'm trying to "cope" with? That American AI companies are overleveraged and some of them are going to go bankrupt? I've been predicting that for months. Why do you think I would need to cope with the idea?
In reality just like every major market runup there's gonna be a consolidation of the industry and inevitably it will cause many companies to fail and critics will point to that and go "see it was a bubble!" while completely ignoring the several dozen new companies that will rise out of it. It's the 90s all over again.
> critics will point to that and go "see it was a bubble!" while completely ignoring the several dozen new companies that will rise out of it.
That several dozen new companies came out of it has nothing to do with whether or not it was a bubble. It was, and the effects of it popping were bad (including the effect that its popping led to consolidation). Nobody should want a repeat of that.
Aren't the models from Anthropic and OpenAI simply the distilled work of everyone else who ever put their work online, or in books?
Why is their distillation okay, but other distillations not ok?