The creators of Slay the Spire have talked about how they gathered statistics on player runs and used that to balance the game. Does anyone know other games that have done this? (Especially single player games.)
Evan Debenham, creator of Shattered Pixel Dungeon [1], gave a talk about how he makes balance and other changes to the game based on analytics gathered from the game’s players [2]. This is used to determine the relative strength of different aspects of the game by seeing how a player’s engagement with a particular aspect correlates with win rate.
This, of course, needs to be checked by community feedback on changes to the game because perfect balance doesn’t always equal fun!
The Fireball spell in D&D is a third-level spell, which means a spellcaster can get access to it at level 5. Compared to other third-level spells, and the power curve as a whole, fireball is overpowered since it does a large amount of damage over a large area, but it exists in the space it does because it's such an iconic and evocative spell/ability that feels so fun to use when it really hits that the designers want people to take it and use it.
That is to say, the spell is so fun to use (which I can confirm) that it's worth being unbalanced if it means more players get to experience it.
I believe that’s very common. I worked at a larger casual game studio and we had Google analytics at the time in the game client. You could easily see where people got stuck based on those and theyd often inform what we did
Sometimes the ability was changed to make it harder or easier to use, or to remove some OP side-effects (e.g. shooting laser would slow you down too much, which in this game it's also an advantage, as you can brake and avoid collision using the laser).
Similar in concept, the designers of subnautica talked about mapping player feedback within the game world and using that to make changes: https://youtu.be/fkjY_R7zQsM Unfortunately I don't have a timestamp because I watched this a long time ago.
It's probably not as rigorous as what you're thinking of but the devs of DCSS have cited online win rates of certain combinations as the impetus for balance changes before.
This, of course, needs to be checked by community feedback on changes to the game because perfect balance doesn’t always equal fun!
[1] https://shatteredpixel.com/
[2] https://youtu.be/NMWQIdfCgQg