[qeternity]

> LLMs are text in -> text out, and you can drop in a new LLM and replace them without changing anything else. If anything, newer LLMs will just have more capabilities.

I don't mean to be too pointed here, but it doesn't sound like you have built anything at scale with LLMs. They are absolutely not plug n play from a behavior perspective. Yes, there is API compatibility (text in, text out) but that is not what matters.

Even frontier SOTA models have their own quirks and specialties.

[msgodel]

When I've built things with them I've mostly considered the quirks defects.

Kind of like how httpds will have quirks but those aren't really a good thing and they're kind of plug and play.

[stavros]

What kind of quirks have you seen that the next model wasn't better at?

[dcre]

A simple example would be when models get better at following instructions, the frantic and somewhat insane-sounding exhortations required to get the crappier model to do what you want can cause the stronger model to be a bit too literal and inflexible.