Ask HN: How would you know if an AI model has been nerfed?

[gitgud]

If consumers are just calling a model in the cloud, what’s actually enforcing that the model is not just a dumbed down cheaper model?

What’s stopping anthropic/openai from just running a cheaper LLM model based on how difficult the question is?

[verdverm]

To detect, you need evals

To have guarantees, contracts

When I use the API, I specify the model. They may use a quant'd version, but they aren't going to change the underlying model (caveat that I'm calling the quant and not the same model in this context)