[tristanj]

No, it's a flawed conclusion.

Claude Fable was publicly available for 72 hours early June. Moonshot more than enough time to prepare infrastructure, gather their preferred distillation data from Fable, and complete post-training well K3's mid-July launch.

[JumpCrisscross]

> Moonshot more than enough time

Genuine question: do you have a source for how long distilling Fable would take with preparation?

[tristanj]

Moonshot already has at least several million exchanges distilled from Claude that they obtained over the past year https://www.anthropic.com/news/detecting-and-preventing-dist... . So they have the infrastructure already set up to do this. Re-running their existing distillation suite on the new Fable endpoint would be trivial.

Given that Fable was available for 72 hours back in June, I asked GPT-5.6 Sol to estimate how many accounts are needed to generate 1–2 million exchanges with Fable within that timeframe. It concluded it is achievable with only a few hundred accounts.

Here's GPT-5.6's conclusion:

Under a deliberately simplified, compliant planning model, a Max 20x account could produce approximately 1,944 to 5,832 standard exchanges during 72 hours when Fable 5 use is restricted to 50% of the modeled subscription capacity. The central planning estimate is 3,888 exchanges per account. The 1 to 2 million exchange target is therefore reachable in the central case with roughly 257 to 515 accounts.

And the analysis: https://markbin.net/s/pd_9DpYsQr9/sh_vxwgtYdQ?sig=9552292ab3...

[amluto]

As far as I know, 100% of those Fable interactions would have had encrypted reasoning blocks, so distillation would be distinctly nontrivial even if the data were somehow available.

[tristanj]

Encrypted reasoning traces don't prevent distillation. You only need the input prompts and final output responses to distill capabilities. As Anthropic explained in their distillation report (linked above), Moonshot AI already collected millions of session traces, covering:

* Agentic reasoning and tool use

* Coding and data analysis

* Computer-use agent development

* Computer vision

Hiding the internal CoT blocks stops you from training on internal reasoning traces, sure, but it does nothing to prevent standard input-output distillation.

Plus, the CoT blocks weren't even removed/hidden completely. They're still visible, just in summarized form. Raw CoT was replaced by summarized CoT, and summarized CoT still has distillation value.

[amluto]

How would you propose to distill Fable-style input/output pairs without the CoT?

If you use them as SFT input, you’ll be trying to train a model to predict the post-reasoning output without any reasoning, and this seem very unlikely to work at all with the size of model that Kimi produced and the complexity of Fable’s output. You can’t really “RL” with them because they would be so far off policy that there would be nothing to reinforce. I suppose you could feed input/output pairs to a teacher model and attempt to generate reasoning traces, but it seems like some wishful thinking would be required to get anything even close to as good as Kimi K3 out.

Maybe Kimi used these traces to generate RL gym-style problems and somehow produced an evaluator based on the outputs? They would not have had a lot of time in which to do this, and the learning style would not even remotely resemble that which Anthropic used to train Mythos/Fable in the first place.

But what do I know? I’m not an expert here.

[tristanj]

Anthropic removed raw CoT from all models since Sonnet 3.7 (released February 2025), all Opus 4.X models have censored CoT, yet Chinese labs continue to distill from them, proving this restriction wasn't an insurmountable hurdle.

There are some outputs that are highly valuable and cannot be censored, such as tool API calls or agentic tool usage, which is precisely what Moonshot was accused of harvesting/distilling.

Tool call harvesting was enough of an issue that Anthropic began inserting spurious tool calls when they detected distillation attempts, trying to poison the distilled data.

There is certainly more data the labs are training off of, but that's difficult to know without insider information.

[boesboes]

What do you base that 1-2 million on? Sounds like complete horseshit imo

[tristanj]

It's based on Anthropic's press release where they reveal distillation of their models by Chinese labs: https://www.anthropic.com/news/detecting-and-preventing-dist...

Moonshot AI already distilled over 3.4 million exchanges; I reached 1-2 million exchanges assuming they would like to augment or improve about half of their existing (distilled) dataset.