[oceanplexian]

I’ve been using them as a customer and have been fairly impressed. The thing is, a lot of inference providers might seem better on paper but it turns out they’re not.

Recently there was a fiasco I saw posted on r/localllama where many of the OpenRouter providers were degraded on benchmarks compared to base models, implying they are serving up quantized models to save costs, but lying to customers about it. Unless you’re actually auditing the tokens you’re purchasing you may not be getting what you’re paying for even if the T/s and $/token seems better.

[dlojudice]

OpenRouter should be responsible for this quality control, right? It seems to me to be the right player in the chain with the duties and scale to do so.

[teruakohatu]

> many of the OpenRouter providers were degraded on benchmarks compared to base models, implying they are serving up quantized models to save costs,

Do you have information on this? This seems like brand destroying for both OpenRouter and the model providers.