[yorwba]

> To test for possible contamination, I tried the same prompts without attaching the sample translations and Claude failed and refused to answer, saying that it is unfamiliar with the Circassian language.

This doesn't indicate that Claude is unfamiliar with Circassian, only that Circassian is sufficiently rare that refusing to answer is a plausible response.

The language is not that obscure in the grand scheme of things, there's a Wikipedia article explaining the grammar https://en.wikipedia.org/wiki/Kabardian_grammar which is definitely in Claude's training set, probably alongside a few hundred linguistics papers and a bunch of monolingual data.

If you measured the performance for different numbers of initial translation examples, I suspect that there will be a sudden jump at the point where Claude stops refusing to even try, and after that additional examples will only marginally improve the output.

[abdullahkhalids]

If all those resources (Wikipedia article and papers) exist, then they are surely in GPT-4's training set as well. So clearly there is a difference between the capabilities of Claude and GPT-4.

[thorum]

If this were correct, it would be a bit less impressive than OP’s claim, but still a monumental leap forward for translation of low-resource languages - a task which all previous LLMs fail at.