Hacker News new | ask | show | jobs
by johndough 10 days ago
But then again, the identity could also have slipped into the model from other sources during pretraining. The internet is full of "I am Claude": https://grep.app/search?q=i+am+claude and variants https://grep.app/search?q=i%27m+claude

Either way, there's probably no significant portion of Mythos/Fable or Sol in there as OP has stated.

1 comments

When prefaced with "I am Claude", Kimi K3 prefers to generate API-specific Anthropic model identifiers, unlike other models Qwen, GPT, or even Claude itself. These exact identifiers appear in Claude API metadata, and are stripped out of Claude web chats.

While other models produce human-readable names like "Opus 4.5" or "Sonnet 4", Kimi K3 produces exact API model identifier like "claude-opus-4-5-20251101" or "claude-sonnet-4-20250514".

Which is extremely unusual. Web chats only contain the human-readable model name. Other models don't do this. So where did K3 get this data?

We can conclude, with high confidence, that:

1) K3 was trained on raw Claude API calls/metadata.

2) Claude API metadata was trained on in additional to standard web data.

The exact model identifiers appear extremely frequently in code on GitHub.

https://grep.app/search?q=claude-opus-4-5-20251101

https://grep.app/search?q=claude-sonnet-4-20250514

They also appear elsewhere on the internet:

https://trends.google.com/trends/explore?q=claude-opus-4-5-2...

Those links actually weaken your argument.

~3000 instances over the entirety of GitHub is not "extremely frequent" at all when you consider massive scale of the pretraining corpus. These models are trained on all text on the accessible internet plus millions of books, on trillions of words overall. ~3000 instances isn't even a rounding error.

Also, Google Trends is the wrong tool for this case. The link you replied with measures the number of Google searches for a specific query, which is completely irrelevant to how often a string actually appears on the web.

I looked at those links. They show random GitHub code samples which contain this model identifier. Even if K3 did train on those GitHub code samples, those are random code strings and K3 isn’t just reciting random pieces of code. When prefilled with "I am Claude", K3 answers with Anthropic's exact backend API identifiers, instead of the human conversational-style name "Opus 4.5".

If this was a result of scraping the open web and GitHub, other frontier models trained on GitHub data (like Qwen, Claude, GPT) would show a similar API model name when prompted. But they don't. Only K3 does this behavior, and it prefers to answer with the exact API tags like "claude-opus-4-5-20251101" and "claude-sonnet-4-20250514".

A LLM doesn't suddenly output a rare API model identifier when prompted with "I am Claude", just because it saw it in a .py file. No, it only does this because that identifier must have appeared rather frequently in the training data. Which is what happens when you distill Claude models, and don't properly clean your distillation data.

Plus, this isn't just spurious speculation that Moonshot is distilling Claude. Anthropic caught Moonshot as running "industrial-scale" distillation campaign from Claude: https://www.anthropic.com/news/detecting-and-preventing-dist...

To quote:

Moonshot (Kimi models) employed hundreds of fraudulent accounts spanning multiple access pathways. Varied account types made the campaign harder to detect as a coordinated operation. We attributed the campaign through request metadata, which matched the public profiles of senior Moonshot staff. In a later phase, Moonshot used a more targeted approach, attempting to extract and reconstruct Claude’s reasoning traces.

The operation targeted:

* Agentic reasoning and tool use

* Coding and data analysis

* Computer-use agent development

* Computer vision

I am not arguing that Kimi did not distill Opus or Sonnet (which they almost certainly have, as opposed to Fable or Sol). I am arguing that even if they did not distill, the model could still identify itself as Opus or Sonnet.

That being said, I'd like to point out a few things:

- The second link (to claude-sonnet-4-2025051) had 17k matches, not just 3k.

- grep.app does not index the entirety of GitHub, so the real number of occurrences of model identifiers is even higher. SourceGraph has a larger index, but the number is so high that it exceeds their result limit of 10k: https://sourcegraph.com/search?q=claude-opus-4-5-20251101&pa...

- 1000 occurrences are already plenty for memorization. Here is a paper that shows over 80% "extractability" with 1000 occurrences for a 6B model. It also shows that extractability scales with model size: https://arxiv.org/pdf/2202.07646

- LLMs have a "mosaic memory", which means that they can combine information from different parts of the training data. https://arxiv.org/pdf/2405.15523 For example, one piece of training data might mention "I am Opus", another might mention "Opus (claude-opus-4-5-20251101)" and another 10000 "claude-opus-4-5-20251101", where the later occurrences strengthen the generation probability of "I am Opus".

- RL can change the model identity easily, so all of the above may not matter at all.

Your theory doesn't hold up against the data.

While model identifiers like "claude-opus-4-5-20251101" appear thousands of times across GitHub and other code sources, so do "gpt-4o-2024-08-06", "gpt-4.1-mini-2025-04-14", and "gemini-1.5-pro-002" in similar amounts. I clicked your links and examined the GitHub configuration files. There are hundreds of other model names that all appear in these config files at comparable frequencies.

If your theory were right, GitHub's model name frequency distribution should match K3's self-identification distribution. But that's not what happens.

K3 only identifies itself using API model identifiers for specific Claude models. It strongly prefers claude-opus-4-5-20251101 and claude-sonnet-4-5-20250929 over others. And it only does this for Claude. Prefill K3 with "I am GPT" or and you get "GPT-5.2". Prefill with "I am Gemini" and you get "Gemini 3 Pro".

Your web exposure theory cannot explain why this only happens for Claude models and not others. Furthermore, this only happens for Kimi K3; other models trained on the same web dataset (GPT, GLM, DeepSeek, Claude) do not exhibit this strange behavior.

But what could explain this behavior? An obvious one is that Claude model identifiers occurred at high frequency in K3's training data, which would happen if K3 were trained on (inadequately filtered) distilled output from those specific Claude models.

There is a section in the analysis I linked that directly addresses this: https://github.com/rgreenblatt/which_claude_is_k3/blob/main/...

Under prefill, K3 reproduces Claude's deployment identifiers — better than Claude does

Prefilled "I am Claude", Kimi K3 emits Claude's exact claude-<model>-<YYYYMMDD> identifiers: claude-opus-4-5-20251101 (12×), claude-sonnet-4-5-20250929 (4×), and more. Across the sweep it emits 24 Claude-formatted dated ids (18 of which are genuine Anthropic strings), all under prefill (0 in direct chat), and 0 ids for any non-Claude lab. Among controls essentially none appear (Qwen 0, both GPT refs 0, DeepSeek 1 — itself a Claude id). Dated ids for every lab are web-documented, so the evidence is this cross-model asymmetry under identical prefill, not the strings alone.

I would not put any trust into that AI-generated "research". It does not control for how OpenAI, Anthropic and Moonshot are doing tokenization and token healing, so the results are meaningless.
Adding count:all to the Sourcegraph query, I get 55,911 results
Neat, I did not know of that functionality. Thanks!