[pavpanchekha]

OpenAI used to make Codex-specific models, but they stopped. What I've gathered from interviews and similar is that training two models isn't worth the (small) lift from having a coding-specific model. You're pre-training on everything anyway, and coding RL is reasonably useful for general-purpose models too.

[greyskull]

Interesting. I'd have guessed there would be meaningful opex benefits to serving smaller models.

[mediaman]

What I've heard is that much of the model "intelligence" is a commingled bucket: although you can specialize specific knowledge somewhat, it's hard to specialize advanced reasoning to specific domains because so much of reasoning is a generalized capability that is not unique to, say, coding.

It turns out coding has to do with a lot of the same reasoning needed in math or in legal analysis, even if the grammatical expression is different.

This is less true of lower intelligence tasks. Classification requires a lot less reasoning capacity and so can be much smaller and more specialized.