[zqfm]

Maybe off the wall, but my first thought is certificate revocation. Could there be (or is there already) a semi-centralized database of research that should no longer be cited? With maybe a dependency graph of research that maybe needs to be revisited?

[quercusa]

There's something quite like this in US law. If you are citing a case in a legal brief, you need to make sure it hasn't been overturned along the way, so you 'Shepardize' it. From Wikipedia [0]:

Shepard's Citations is a citator used in United States legal research that provides a list of all the authorities citing a particular case, statute, or other legal authority. The verb Shepardizing refers to the process of consulting Shepard's to see if a case has been overturned, reaffirmed, questioned, or cited by later cases.

[0]: https://en.wikipedia.org/wiki/Shepard%27s_Citations

[jmnicholson]

We're effectively bringing shepardizing to science at scite (scite.ai). Here's a piece I published on the topic recently: https://www.sciencedirect.com/science/article/pii/S259023852...

[timlod]

Hi! I checked out scite and really like the product- however, I can't seem to find relevant papers in my field (ML&AI), ie., I searched for the batchnorm paper, which there has been a lot of 'reevaluation' of- but it's not available. I suppose at this point you don't have access to (some of) the big AI journals/conferences. Is this something on the horizon?

[em-bee]

this sounds like a good approach. right now every citation is an endoresement, and the choice is to support a publication or to ignore it. there should be a way to make a negative citation, that allows you to declare that the referenced publication is contradicting. as a result there would be two citation counts for each publication. like upvotes and downvotes. i believe contradicting citations do happen now, but the fact that they are contradicting can't be seen from the citation count.

[d110af5ccf]

> right now every citation is an endoresement

This is a valid point, but it's important to note that it only applies when using aggregate citation counts as a rough proxy for importance within a field. That's useful for prioritizing what to read but not for assessing the validity of any given result (of which there are often quite a few, at least in the life sciences).

Within a given paper, it's not at all uncommon for the authors to explicitly call out some detail from another study as being incorrect in their view. That doesn't mean that they necessarily agree or disagree with the rest of the cited work though.

What I'm getting at here is that negative citations would likely be far too coarse to be useful in practice. It's relatively rare that a paper outright disagrees with an entire work.

[probably_wrong]

I've thinking how to make this idea work, but the more I think about it the less I think it's viable.

First, you would be centralizing the ability to "cancel" papers. One bad member of the board could wreak havok. Simultaneously, people in charge of the database would be constantly bombarded by pleas (not all of them honest) to revoke this and that researcher's paper. So it would be bad for both sides.

You could get away with something like Retraction Watch, but you'll always be behind - you could keep track of which papers were retracted (which is already a lot of work), but not every paper that needs to be revised is retracted.

I guess figuring out a perfect system for deciding which ideas are right is hard.

[worldsayshi]

You point out the flaws of one potential implementation of such system and reject the whole concept? I think it's conceivable that there is an implementation of such concept that would work better than not having it.

Example off the top of my head: Don't have one board decide which papers are cancelled. Have multiple. Have git repos with lists of cancelled papers. Have a process for finding which papers are flawed due to citing cancelled papers from a particular paper. Have browser extensions that tells you if the paper you're opening is carrying such flaws. Allow user of such extension to pick which board they trust. Have the extension link to a discussion about why the paper is cancelled. Give ability to challenge such claim in the open. Etc etc.

I'm sure many more such ideas can be found. The problem is probably rather funding of such system or if whoever funds it would have the right incentives to begin with.

[dougmccune]

This is part of what https://scite.ai/ is doing. They categorize citations as supporting, detracting, or neutral. The theory being you want to know if you’re citing something debunked. Scite also flags citations of retracted papers (and a Twitter bot that tweets when a new paper is published that cites a retracted study). And I think they have zotero integration (?)

I have no affiliation other than I have met the founder and think the product is cool.

[eecc]

WOT signed endorsements. Back when I had time I drafted a P2P application to share papers and execute peer review via GPG based digital signatures. Signed metadata - at least that was the plan - allowed communities to endorse, retract, flag spam in a distributed opt-in manner.

It's rotting on GitHub, never managed to drum up enough interest...

[CuriousSkeptic]

I’ve been toying with a similar idea but as a means of crowdsourcing a current best understanding of what science suggest. Primarily to weed through all the wild nutrition recommendations floating around into sensible advice based on what can actually be supported. I’m sure the idea extends to other areas.

In my mind it would be some reddit/Wikipedia kind of thing where advice and understanding would be debated using some formal language elements based on rdf or whatever to build up a knowledge graph usable inside applications.

[sam_lowry_]

Sounds like a great idea. WOT did not work for general public, but scientists are very different.

[mjevans]

Things that work for the general public need to be far less technical and way more automatic; from an end user experience perspective.

WoT could work exceedingly well for a decentralized replacement of Facebook... though at that point there's also the issue of competing against a walled garden with moats and most of the population already inside.

[heyitsguay]

Maybe instead of a centralized repository, individuals could be given the ability to create their own partial sets of recommendations that could be aggregated by a user. Instead of trying to globally solve trust and designation of expertise, you explicitly define the group of people whom you designate experts and whose assessments you trust.

[ahelwer]

Science's decentralized nature is a strength, not a weakness. Do you really think such a database would be immune to coercion from, say, Monsanto or Chevron?

[SubiculumCode]

building that in Zotero would be great. I all ready like that they flag retracted papers.

[domenicrosati]

Done https://github.com/scitedotai/scite-zotero-plugin

[SubiculumCode]

you sir/madam, are awesome.