[ijk]

Inference using an API costs money. Not a lot of money, per million tokens, but it adds up if you have a lot of tokens...and some of the obvious game uses really chew through the tokens. Like chatting with a character, or having the NPC character make decisions via reasoning model. Can easily make the tokens add up.

Games, on the other hand, are mostly funded via up-front purchase (so you get the money once and then have to keep the servers running) or free to play, which very carefully tracks user acquisition costs versus revenue. Most F2P games make a tiny amount per player; they make up the difference via volume (and whales). So even a handful of queries per day per player can bankrupt you if you have a million players and no way to recoup the inference cost.

Now, you can obviously add a subscription or ongoing charge to offset it, but that's not how the industry is mostly set up at the moment. I expect that the funding model will change, but meanwhile having a model on the edge device is the only currently realistic way to afford adding an LLM to a big single player RPG, for example.

[K0balt]

You release the game with a variable in game experience. If the player has two 4090s chugging away, she runs everything locally. If he’s got an rx480, a barebones 1b model or a subscription for the nicer AI NPCs, which can open up AI driven side quests (adding minor content within having to write it). Include a “free” month when you register the game.