[kylecazar]

Yes. I do wish there were benchmarks for specific tech stacks. I.e, if I have an Elixir/Phoenix project, which model performs best (idiomatic, etc.) in 2026?

Of course it will be somewhat subjective. And I can hang around those communities for opinions. But it might be useful in a world where it's impractical to constantly compare them all, and it varies pretty widely.

[pocketarc]

This absolutely should be a thing, but it'll have to be a per-community thing, them building their own dataset and creating their own evals (similar to how people do for production workloads).

Although:

"(idiomatic, etc.)" I don't think that should be part of the aim (or it should be under-weighted), because... you can just provide guidance on how to do things more idiomatically, rather than depend on that knowledge already being encoded in the model. I'd be more curious about verifying that it can work through gnarly bugs / features in an Elixir codebase. After all, what use is a model that by default does everything idiomatically if it can't figure out some small concurrency bug.

[andai]

> you can just provide guidance on how to do things more idiomatically, rather than depend on that knowledge already being encoded in the model

I spent some time a while ago comparing various Claude models to various Chinese alternatives. The main difference I noticed was that Claude's code tended to be much more readable and much more elegant.

I found myself asking Sonnet to clean up code written by other models on a very regular basis. (Maybe that's subjective, but I have a feeling it would win on some objective measures of code complexity as well.)

Eventually I realized I could just prompt the Chinese models to give me elegant code. If you don't tell them that they will give you pretty mediocre code by default. (This has improved a little with recent releases.)

I thought that was pretty funny, but then I remembered, it used to be that way with image generation models too!

Back in the day, you had to add "+good -bad" to the prompt, otherwise they wouldn't know you wanted the art to look good, and they would give you all sorts of crap! (Because they've been trained on all sorts of crap. Well, most of the art online is pretty crap, and that goes for most code too.)

[0xCAP]

It should become a win/win mechanism where communities are rewarded for high quality, human driven validation of LLMs, and vendors gain for bragging rights about their models being highly-skilled-human-approved. Still don't know why this isn't becoming a thing.