[0xDEAFBEAD]

That's fair. Maybe you could even send the user an email if you detect a new model release or pricing change which handles their workload for cheaper at comparable quality, to notify them to investigate.

[iamtherhino]

That's a good idea-- then give them a link to "replay last X inferences with model ABC" so they can do a quick eyeball eval.

[0xDEAFBEAD]

Sweet, maybe you'll like my other idea in this thread too: https://news.ycombinator.com/item?id=43929194