I like the idea of preserving the best comments as some form of blog. TheBestOfHN.com might be worth starting up by just compiling all comments with > 200 karma?
Those scores are not so interesting by themselves in that they were calculated based on votes at the time and aren't meant for comparing comments on different articles. (The top comment on a high-scoring article will automatically beat the top comment on a low-scoring article)
I would rather like to see what Dan did scaled up 1000x or so: that is, have experts pick out a few of the best comments over the very long term with a number of quality criteria in mind.
Like it or not individual people have characteristics that are mostly stable over time: they are more or less smart, mean, wordy, honest, loyal, interested in different things, make different mistakes, etc. Profiling the author is worthwhile.
The article though is a problem not a solution in terms of sampling: consider that people talked about topic A in 200 articles that hit the front page in the last 5 years; if you put the comments from those 200 articles in pool B and rank the articles in pool B and pick up the top ten you would get the "pure gold".
is this actually feasible - if we push it beyond HN then it becomes trying to crowd source wisdom or deep expertise across human experience. The Anti-Social-Media-Algorithm?
The case of Wikipedia shows that efforts like that can be scaled but it's hard to find more examples.
A counter example is the old DMOZ directory which was a continuation of the tree-style Yahoo web directory.
That was subdivided into sections that were managed by "experts" (admins appointed to the sections) but the admins were lazy, corrupt or both. Even if they had been doing their jobs they would have been overwhelmed by hardly relevant "content marketing" submissions for web pages that are "200 OK", hot keywords, and a scam.
You have to draw from a spam-free pool.
For instance there is a data dump for stackoverflow
personally i think stackoverflow is a junk web site full of more wrong answers than right answers and wouldn't it be nice to pick just the best answer to the question and not see the code sample at the top that didn't work that the original poster was asking about?
If you are a real programmer, I mean you are shipping, you write a unit test, if it doesn't work they are gonna send it back to you, you may learn the hard way that answer #1 is not right (the compiler said so!) and scroll through a lot of half-baked discussion before you find that answer #7 passed the unit test, the acid test, all the other tests -- you are an expert and your opinion is worth much more than the "crowd".
If "Road Scholars" like us were getting our experience fed back into Stack Overflow how different would it look?
I would rather like to see what Dan did scaled up 1000x or so: that is, have experts pick out a few of the best comments over the very long term with a number of quality criteria in mind.
Like it or not individual people have characteristics that are mostly stable over time: they are more or less smart, mean, wordy, honest, loyal, interested in different things, make different mistakes, etc. Profiling the author is worthwhile.
The article though is a problem not a solution in terms of sampling: consider that people talked about topic A in 200 articles that hit the front page in the last 5 years; if you put the comments from those 200 articles in pool B and rank the articles in pool B and pick up the top ten you would get the "pure gold".