I'd say they have a "Tier 2" inference stack, but that'd be Apple Silicon and AMD: their well below that.
In part because unlike those, no one can get their hands on the hardware, so they miss out on a massive amount of free development, testing, etc.
At this point there's years of complaints about open weight models performing worse on their platform, tool calling makes it especially easy to tell since it's so sensitive to these issues.
Simply, some of us have used the service and have not noticed special issues - so, if one states «so many random errors, so many silly quirks», it's best to substantiate.
How, systematically speaking? I would need a properly structured embeddings DB over HN (and possibly a few other sources). I am not aware of any satisfactory (reliable) RAG available. I have found this:
which seems to be off at the moment. So, when John says "Jake I experienced as so itchy", I may expect some sauce from the speaker, not the tiniest crystal of the juice, if my takeaway is not just to be "John had a suboptimal experience with Jake". (And itchingly, "What's "itchy"?" ("what "random errors"?", "what "silly quirks"?")).
I'd say they have a "Tier 2" inference stack, but that'd be Apple Silicon and AMD: their well below that.
In part because unlike those, no one can get their hands on the hardware, so they miss out on a massive amount of free development, testing, etc.
At this point there's years of complaints about open weight models performing worse on their platform, tool calling makes it especially easy to tell since it's so sensitive to these issues.