[Mandelbug]

Unfortunately, while showing what we would consider "intuitively bad data" may seem like a bad idea, only actually empirically testing it can we measure and quantify the exact impact that bad data (either shown purposefully or accidentally, or unknowingly) will have on users.

It may have in fact turned out that what we intuitively think as bad data results in better matches or better experiences. I think experimenting is worthwhile, so long as it is done in the open as they have been doing.

[nobodysfool]

I remember reading about this concerning Netflix. They used to have a competition on who can come up with a better recommendation algorithm. They eventually decided not to implement the best one because what they were getting was good enough, and since most people were streaming instead of getting a dvd, the more recommendations the better. They could just try and if they don't want to see the whole movie, rate it and instantly pick another.

[thelonelygod]

I'm pretty sure they didn't implement it because it was an extremely impractical solution.

Edit:

"We evaluated some of the new methods offline but the additional accuracy gains that we measured did not seem to justify the engineering effort needed to bring them into a production environment." - https://www.techdirt.com/blog/innovation/articles/20120409/0...

[nobodysfool]

>Streaming has not only changed the way our members interact with the service, but also the type of data available to use in our algorithms. For DVDs our goal is to help people fill their queue with titles to receive in the mail over the coming days and weeks; selection is distant in time from viewing, people select carefully because exchanging a DVD for another takes more than a day, and we get no feedback during viewing. For streaming members are looking for something great to watch right now; they can sample a few videos before settling on one, they can consume several in one session, and we can observe viewing statistics such as whether a video was watched fully or only partially.

I wouldn't say that this makes you wrong, but I'd say you are partially right.