[simonw]

OpenRouter rankings frustrate me, because they show the total number of tokens but they provide no indication of how many unique users a model has.

Which means if a surprise model tops the leaderboard one week we can never be sure if it was because a single whale user pushing billions of tokens a day switched to it, or if it represents a genuine community trend towards that model.

[numlocked]

(openrouter co-founder here)

Yeah we should do something to indicate cardinality. I can share that there can often (I'm talking generally; not related to this model in particular) be e.g. a very large app that can be pushing a lot of volume. But in almost all cases that app has a large number of end users. Hypothetically, for instance, would Cursor be consider one user, or millions?

Will think about it! Thanks for the feedback.

[simonw]

I'd consider Cursor one user because it's one entity that made an editorial decision about which model to make available to their own community.

If you treated Cursor as millions of users it might look like millions of people independently chose a new model when actually it was Cursor making the choice for them - and the thing I care most about is how many choices were made that selected a model and put it above the others.

[dotancohen]

An alternative viewpoint is that the single choice made about switching the Cursor model was done after extensive testing by a competent and experienced team. Whereas my naive self choosing a model to play with this week is far less a signal to others that the model is fit for purpose.

[minimaxir]

One idea I had was to count # of distinct API keys that have spent atleast $100 (number's flexible), which would be enough to provide guidance on if the traffic is from a single power-user.

In the Cursor case which is BYOK, that would count as distinct API keys.

[martinald]

Hi! Big fan of OpenRouter and the data you provide. It'd be awesome if you would consider providing volume of tokens per hour, mostly for my own curiosity as to quite how peaky demand is.

Thanks!

[svantana]

Also, while we're pitching new features to openrouter, I'd like to see a "$ spent" chart, which would remove all these huge freebie spikes. It looks like it would be pretty much dominated by claude.

[senordevnyc]

Agreed. My little solo dev SaaS app’s production pipelines push almost two billion tokens a day.

[senordevnyc]

Haha, I never tire of the AI haters downvoting stuff like this.

Down with reality!!

[daveguy]

Or, everyone finally realizes that token burn is not the same as productivity. Maybe they just down voted for the questionable spending brag.

[senordevnyc]

We were talking about whether these metrics are meaningful. I was just pointing out that even a tiny one-person company can burn a lot of tokens.

As to whether the token spend is questionable, the number I quoted is for my production AI pipelines, not for coding. And my customers (and profit margin) seem to think the spending is valuable.

[dotancohen]

Questionable spending aside, GGP is providing information about how a specific metric may not measure what people think it measures. There is value in that comment.