[pbhjpbhj]

I suspect under closer analysis ones scoring breaks down to be inconsistent - "in retrospect I like tea X better than Y but not as much as Z, but I rated Z lower than Y because I didn't like it as much as P which had a higher rating", if you follow.

[snprbob86]

It seems that, pairwise, it is pretty easy to decide. Maybe one could go further than this and eliminate the absolute scale all together (at least at rating time).

I'm imaging a UI which asks you to pick a favorite among the item you are viewing and one similar item. You could stop there, or ask repeatedly with new comparison items until the viewed item's position on the absolute scale is unambiguous. The user could provide some rating data with just a single binary decision, but some ajax-y fade out/in of another pair could enable further ratings if they desired.

[m_eiman]

If you do that, the Elo rating system is a good place to start algorithm-wise.

http://en.wikipedia.org/wiki/Elo_rating_system

[jurjenh]

Taking this idea further, one could add multiple orthogonal axes (eg sweetness, bitterness, after-taste etc). Then you could rate each tea against others on each axis - either in a star-slider, scatter plot or on several individual sliders. This would allow you to rate teas against each other based on several aspects, and possibly allow recommendations based on how other people have rated teas - eg 'I want a tea that is not-too-sweet, a little bitter with a lingering aftertaste' Then again, it does add more features / visual clutter and possibly complicates things for people...