[RiderOfGiraffes]

You seem to have misunderstood. I'm not saying that the DupDetector is helping to find good, older submissions. The idea is to get help reduce the endless repeats, to provide space, and then, instead of endlessly repeating the same submissions, have a way that people can find the good ones that are already there.

But what running DupDetector has shown me is that the problem is endemic. There is no way that I can see to reduce the rate of re-submissions. People have got very annoyed when the problem has been pointed out. Indeed, many people don't see it as a problem. The very fact that people see an enormous number of posts by DupDetector tells us two things: There are a lot of duplications, and people don't want to know.

Some discussion here: http://news.ycombinator.com/item?id=2013666

So that's fair enough. As I say, I'm terminating the experiment. I've learned that the current level of noise caused by constant re-submissions of the same stories, and often the exact same items, won't stop. So instead I'm concentrating on finding items that fit with PG's stated preference:

    Stories on HN don't have to be about hacking, because
    good hackers aren't only interested in hacking, but they
    do have to be deeply interesting.

(From http://ycombinator.com/newswelcome.html)

On any day, look at the "newest" page and ask yourself how many are really "deeply interesting", how many are amusing diversions, and how many are diverting, but useless. Ask how much you learn from each one. The answer I suspect will be "not much".

Look at the "news" page and ask the same question. The problem is that if enculturated "old hands" don't read the "newest" page, then the "news" page will deteriorate, because the people voting on new items won't be voting the right things for the right reasons.

We see that now. Lots of submissions get votes because they are, in PG's words, "intensely but shallowly interesting."

So I'm going away for a while to see if I can do that. I'll still read stuff, but every time I do I will consciously remind myself that I can get content-free entertainment pretty much anywhere, and that I'm looking specifically for stuff that's "deeply interesting." If I figure out a way to find it automatically I'll certainly come back and share it, although I don't expect I'll succeed. It's a hard problem, and I'm certainly no smarter than others who have worked on it before me.

Currently the best option seems to be to have a small community of like-minded people. That's what HN used to be, and I think that despite PG's excellent efforts, it's getting too big to hold together.

But I'll work on it, and I'll certainly share any useful findings. If there are any.

[user24]

Thanks for the reply. I understand a little more about what you're trying to achieve now. And actually I agree with you.

> Lots of submissions get votes because they are, in PG's words, "intensely but shallowly interesting."

Agreed. The "High quality typefaces" (nee "25 new free high quality fonts") post is a prime example imho. And yet trying to point this out is frowned upon massively - complaining in any way that a popular submission isn't hackerly or doesn't fit is a surefire way to get downvoted into oblivion (and, I guess, whinging isn't going to get anyone anywhere, so fair enough).

Perhaps we need more emphasis on flagging articles as opposed to upvoting them.

A HN style site where you can only downvote articles, and the least downvoted rise to the top would be interesting.

[Mz]

But what running DupDetector has shown me is that the problem is endemic. There is no way that I can see to reduce the rate of re-submissions. People have got very annoyed when the problem has been pointed out. Indeed, many people don't see it as a problem. The very fact that people see an enormous number of posts by DupDetector tells us two things: There are a lot of duplications, and people don't want to know.

It looks to me like this is an issue rooted in friction between those folks who spend a great deal of time here and those folks who spend less time here. Those folks who are here a great deal see all the duplicate submissions and find them annoying -- "BTDT, got the t-shirt, can we puhleez move on??". Those folks who spend less time here see it as a new submission or submit it without having any idea it was previously submitted. It sounds to me a little like the running battle you see on some email lists between folks who get digest (and have hissy fits about people not trimming replies) and folks who get individual emails (and feel that trimming replies is too time consuming and loses too much context, thus hurts communication).

I'm sorry you are frustrated by this. And I'm sorry if this issue contributes to "evaporative cooling" of this site (ie by you and others leaving -- my general impression is that you are a significant contributor and that the folks most annoyed by duplicates are usually also important, valued members). But I don't think that vilifying individuals who have posted something in good faith is a constructive solution to a systemic problem.

Please do let us know if you come up with something that might effectively address this issue.

Peace. Thanks for all you have contributed. I imagine many people here will miss you if you leave.

[user24]

> It looks to me like this is an issue rooted in friction between those folks who spend a great deal of time here and those folks who spend less time here. Those folks who are here a great deal see all the duplicate submissions and find them annoying

Yeah I think that's a lot to do with it. I've noticed time and time again with online communities that the longer you use it, the more likely you are to start complaining that it's not as good as it used to be. It happens everywhere, and the problem is almost invariably you not it.