[HarHarVeryFunny]

It's obviously a question - did they just train for 10x as long due to having 10x fewer GPUs, but then that spoils the claim that they distilled Fable which was only recently introduced. Now doubt they did use some training data generated from older US models though.

[upbeat_general]

This is just not how it works - it is perfectly plausible (in fact the most likely) that pretraining was well finished by the time Fable was released. A bit of extra distillation takes far less compute.

Moreover, it’s very plausible (and expected) to use multiple clusters and GPU types for RL rollouts which could very well not be included in this count.

No part of this pipeline is fixed in stone.

[HarHarVeryFunny]

OK, but that doesn't address the question of what can be inferred from Moonshot apparently having access to far less compute then the western labs. To what extent does this reflect the need for less compute due to the (published) architectural efficiencies of their model, and to what extent that they just trained for longer (esp. pre-training - 80% of total compute, perhaps)?

I think the word "distillation" needs to be used a bit more selectively here. If their pre-training run was complete before Fable was released that implies that ZERO Fable data went into the base model. Perhaps the timeline allows for a few weeks at best of incremental post-training on some limited amount of Fable data, but calling this "distillation" seems a bit dramatic especially given the redacted outputs that would have been available. A more factual speculation would just be that they may have had time to post-train using a limited amount of Fable output in some fashion (LLM as judge? SFT? Who knows ...).

[infecto]

Relatedly I also wonder how much the orchestrated distillation bot nets that Anthropic uses against China are real. Not saying it does not happen but I have never seen dating showing how pervasive it is and I always imagine other western labs would absolutely be doing the same.

[theblazehen]

There's significant amounts of traffic going through transfer stations [1], which is then re-sold to the model labs. For a while the DeepSeek API was serving Fable as the DS 4 Pro model even. [2]

[1] https://www.chinatalk.media/p/how-to-buy-cheap-claude-tokens... [2] https://xcancel.com/synthwavedd/status/2078514339552628880