[milkshakes]

assume you are a "second class lab" and you are in fact making progress by distilling the results of the frontier labs' efforts.

what is the end game for this strategy?

if the frontier labs shut down, or stop releasing to the public, and there's noting left to distill, how will you progress?

[InsideOutSanta]

This line of thinking makes no sense because it assumes that labs that distill from frontier models are doing nothing else. It's the classic "the Chinese can only copy" mentality, and it's going to end poorly for American companies.

I'm pretty sure that all labs are distilling each others' LLMs, maybe apart from Anthropic and OpenAI. It would be stupid not to do it, because it's cheap and effective. But that's not the only thing they're doing. If you think K3 and GLM-5.2 got this good only from distilling frontier models, you're not paying attention to Chinese labs' publications.

[milkshakes]

i never assumed that, and i do keep up with the publications. i'm also not saying it's a dumb thing to do! what i am saying is that empirically, it appears that distillation of a more advanced model is a required first step for them to train a borderline competitive, cheaper model. in effect, their training is subsidized by the frontier labs.

if this were not the case, then we would be observing chinese models that far surpass frontier models in capabilities, rather than "almost as good, but much cheaper", and we would be having a very different conversation. what happens to these efforts when the subsidy is cut off?

[InsideOutSanta]

> empirically, it appears that distillation of a more advanced model is a required first step

I see no evidence for that.

> if this were not the case, then we would be observing chinese models that far surpass frontier models

It's pretty clear that the primary reason for the difference is budget and compute availability. Chinese labs have at least an order of magnitude less money than Anthropic and OpenAI.

> what happens to these efforts when the subsidy is cut off?

They will continue making progress as they do now, minus the benefits of distillation.

[milkshakes]

https://www.anthropic.com/news/detecting-and-preventing-dist...

Moonshot AI Scale: Over 3.4 million exchanges

The operation targeted:

Agentic reasoning and tool use Coding and data analysis Computer-use agent development Computer vision Moonshot (Kimi models) employed hundreds of fraudulent accounts spanning multiple access pathways. Varied account types made the campaign harder to detect as a coordinated operation. We attributed the campaign through request metadata, which matched the public profiles of senior Moonshot staff. In a later phase, Moonshot used a more targeted approach, attempting to extract and reconstruct Claude’s reasoning traces.

[InsideOutSanta]

I'm assuming you posted that as evidence for the claim that "empirically, it appears that distillation of a more advanced model is a required first step", but I don't think it is. It's just evidence that Moonshot distills Anthropic's models, which, yes, they do.

[milkshakes]

it is not a required first step for training a model, sure. but that's not what i claimed. what i claimed is that is how they are so significantly _reducing the cost_ of training one! how else do you think they are doing it?

[idiotsecant]

>request metadata, which matched the public profiles of senior Moonshot staff

Translation: we have the machinery in place to identify our users, and actively do so.

[tristanj]

Distillation from a teacher model solves the self-start problem, that is, building a model to the point where it reason coherently. Without distillation, solving self-start is incredibly difficult since it requires millions of high quality training samples. Creating that kind of dataset takes an enormous amount of effort.

Once a model becomes competent enough to perform complex reasoning, a teacher model is no longer necessary. The model can now reason about its own behavior and build a better version of itself through recursive self-improvement (RSI).

Kimi K3 is capable of RSI.

[anon373839]

> Creating that kind of dataset takes an enormous amount of effort

Can you imagine the amount of effort it takes to write 15 trillion tokens worth of art, literature, source code, textbooks, scientific papers, news articles, etc.? No wonder Anthropic just scooped it up and took it for free!

Yet you don’t seem bothered by this. I wonder why.

[IncreasePosts]

There doesn't need to be progress at this point. Some models even from 1 or more years ago are useful for some purposes

[traverseda]

In public with budgets that don't risk destroying the American economy presumably. Yes it may be slower.

[milkshakes]

> with budgets

and what will fund these budgets exactly? inference is cheap, distillation is cheap, training is what's expensive.

[traverseda]

Same people who fund linux kernel development. A coalition of companies that find it useful.

[Jtarii]

Presumably the US military / NSA.

[milkshakes]

the USG/NSA will fund chinese labs? to what end?

[Jtarii]

I was more thinking they would be funding US labs.

[milkshakes]

the question was: what is the endgame for the stated "second class labs" strategy of distilling their frontier competitors then undercutting them on price?