Yes. People who live and breathe your product should absolutely be listened to. Especially when they don't have the super-user tools you do for support (or unconsciously rounding off sharp edges).
It's true. There's probably not a clean rule for when to listen and when not to.
I would propose as a heuristic that for the early stage, when your product relies on true-fans, you at least consider the content of a complaint more and don't just treat it as a signal that something somewhere is wrong. Bringing it back to the OP, I'm sure this is what Card did himself.
Fan-modded games are often great fun if you're seriously into a game. But they're rarely better if you've never played the game.