[nfriedly]

I wonder if a system that assigned weights to each individual user's rating based on that user's rating history could help there - if a user always rates products with 5-stars, then another 5-star rating shouldn't have nearly as much weight as one coming from a user that gives a fairly balanced range of ratings.

I'm not sure if that would actually work better in practice, but it's at least an interesting idea.

[fnordian_slip]

At first your system seemed to me like a solution to bought reviews (mturk and otherwise) and bots.

Then I realized that it would just incentivise bots to add 1-star reviews to random products once their creators figure out this mechanism.

Sometimes these problems make me sad, it could all be so nice and easy if it weren't for these bad actors.

[BearGoesChirp]

It would be a lot more work, but one could check the validity of ratings based on how other users rate things compared to this user.

Say there are 4 games. Most users who rate all 4 rate them similar, except for the 4th game that always gets really low. So 5,5,5,1 is a normal expected rating, but 1,1,1,5 isn't. So 5,5,4,4 from a high rater or 2,2,2,2 from a low rater would be given more weight than a 1,1,1,5. Other things can be added such as weighting a user's ratings a low impact if they have too few scores to determine ratings from.

This reminds me of the problem of determining the answer key to a multiple choice test given only the answers of the test takers.

[bllguo]

unfortunately sample sizes will probably decrease drastically when you're looking at the subset of people who rated 4 specific games, or other such cases

[fnordian_slip]

Yes, that alone is probably enough to make any system like that come with a huge false-positive rate.

I myself only rarely post a review, and I could easily see how - in a vacuum - my star ratings might look like those of a bot.

[splonk]

This does actually happen, possibly for several other reasons. Bought ratings often come accompanied by other ratings, whether bought or not, just to attempt to cloak the bot. This even happens in systems where there aren't any countermeasures being taken to remove bots yet - I was vaguely involved in spam detection in one and saw all sorts of cloaking behavior even before we'd turned on any sort of mitigation.

[nfriedly]

Yea, that's the main reason why I'm not sure it could be made to work.

[Tyr42]

What if you just only say visit restaurants which are well reviewed, and deserve 5 stars? Why should avoiding going to crappy restaurants penalize me when reviewing great restaurants? You're kinda assuming that people are visiting restaurants at random, and should experience the full range of good and bad things, but that's not true, especially for people who rely on existing ratings a lot.

[dagw]

On the other hand most people are only generally moved to bother writing a review for something if it's exceptionally good or exceptionally bad. So someone only giving 5-star ratings could very well be someone who only bothers writing a review if a product is a life changing experience and as such his opinion isn't less valid than that of someone who reviews every little thing he has ever bought.