|
> How do you go about comparing models? I do this often: I'd have a prompt in mind before I manually find the fix for subtle bugs, or plan out how I'd implement a feature. Then, I prompt the models I want to test (with the prompt I originally had in mind) and see if any of them get it right. To my surprise, the Claudes always perform better, and so, it seems like my prompting behaviour is either attuned to them, or Claudes are good at "figuring it out". That said, in my experiments, I've always found that given a "good" prompt (with just the right detail), the top coding models (including from Chinese labs, which I now increasingly prefer) have no discernable difference; in which case, I usually defer to the cheaper & fastest of those models (presently, that's either DeepSeek v4 Flash & Pro, MiMo v2.5 & Pro, MiniMax M3; these are a coding tier below GLM5.2, Kimi K2.7, and Qwen 3.7 Max, imo) as ranked by OpenRouter: https://openrouter.ai/rankings?view=month |