[Alex-Programs]

Hey, I happen to have run into a similar issue with my project!

I've documented a lot of my research into LLM translation at https://nuenki.app/blog, and I made an open source hybrid translator that beats any individual LLM at https://nuenki.app/translator

It uses the fact that

- LLMs are better at critiquing translations than producing them (even when thinking, which doesn't actually help!)

- When they make mistakes, the mistakes tend to be different to each other.

So it translates with the top 4-5 models based on my research, then has another model critique, compare, and combine.

It's more expensive than any one model, but it isn't super expensive. The main issue is that it's quite slow. Anyway, hopefully it's useful, and hopefully the data is useful too. Feel free to email/reply if you have any questions/ideas for tests etc.

[kegs_]

Hey thanks for the reply! Is this "hybrid" method what you wrote in the last line - llm comparison?

[Alex-Programs]

I'm not quite sure what you're asking?

It is in the LLM comparison blog posts, at least the newer ones, though it tends to be on the first line.

[kegs_]

Sorry, when you said hybrid I was expecting something that was partly an llm and partly something else. How did you arrive at your coherence/idiomaticity/accuracy numbers (if you'll forgive me not delving too deep into the website)?

[Alex-Programs]

Hybrid as in a combination of different LLMs. I recommend trying the demo on the site, it should give you an idea of what it's doing. The code is also pretty short.

So those numbers are from an older version of the benchmark.

Coherence is done by:

- Translating English, to the target language, to English

- repeating three times

- Having 3 LLMs score how close the original English is to the new English

I like it because it's robust against LLM bias, but it obviously isn't exact, and I found that after a certain point it's actually negatively correlated with quality, because it incentivises literal, word by word translations.

Accuracy and Idiomaticity are based on asking the judge LLMs to rate by how accurate / idiomatic the translations are. I mostly focused on idiomaticity, as it was the differentiator at the upper end.

The new benchmark has gone through a few iterations, and I'm still not super happy with it. Now it's just based on LLM scoring (this time 0-100), but with better stats, prompting, etc. I've still done some small scale tests on coherence, and I did some more today that I haven't published yet, and again they have DeepL and Lingvanex doing well because they tend towards quite rigid translations over idiomatic ones. Claude 4 is also interestingly doing quite well on those metrics.

I need to sleep, but I can discuss it more tomorrow, if you'd like.