[jsrozner]

Idk if we're automating humans out of the publishing loop as much as rapidly automating the production of crap. I had a very similar experience reviewing for EMNLP recently.

We are nowhere near AI being able to judge the quality of research (in fact, one might reasonably state that even most humans can't really judge the quality of research). Most things in society are not like math: we can't automate (via verification) our way out of noise overwhelming the signal.

Folks are willing to entirely abuse the public resource that is faithful, honest reviewing. (This is unsurprising; the abuse of the commons / public resources has been rising for a long time). There isn't a good solution other than something akin to draconian social scoring to limit access to the reviewing system.

[boplicity]

> draconian social scoring

This is a legitimate question: should people have reputations? Should their behavior be made more visible publicly, both good and bad? How?

In a small contained society, where consequences are more directly affecting individuals immedately, these questions don't need to be asked, because they're inherantly answered. We now are a society with billions of people, and dire consequences sometimes deferred for a generation, or more. Part of our general failing is the lack of good answers to the above questions. For many people, there are rarely negative consequences for causing harm to others, and the rewards can be very great indeed.

[ryandrake]

"Consequences and reputations stemming from one's actions" isn't necessarily draconian social scoring. Without a structure for imposing consequences on wrongdoers, we're not a society, we're just monkeys flinging poo at each other.

[blackqueeriroh]

The problem is that everyone imagines some sort of just and fair arbiter of these things, when the reality is all of the social scoring and consequences and reputations rarely actually stem from one’s actions, and far more often stem from how much money someone has put in someone’s pocket, who someone knows, the color of someone’s skin, or what’s between someone’s legs.

Until we’re actually serious about treating people equitably, (not equally, as that would simply leave the lopsided power structure we have in place) we aren’t getting out of this.

[somenameforme]

I don't understand the desire to argue against this in identity politics terms, which I assume you realize is going to be extremely divisive. There's a far more simple argument that people, independent of ideological framing, would tend to agree upon. And that's what is valuable to one person isn't valuable, to the same degree, to another. For things that are completely illegal, like murder, such a system would be pretty useless because we already have systems in place to punish that sort of behavior and defacto social scoring on it as well.

So this is going to come down to things that aren't usually illegal or 'that illegal', but otherwise affect society. So for instance one group might want to reasonably punish the directors of a company that causes emissions. Another group might want to reasonable reward the directors of a company that creates a large number of desirable jobs. And even in this one example you immediately end up with a weird scenario. Is it okay to pollute as long as you make enough jobs? Who gets to decide that?

There's no need for cartoon villainy for this to be a bad idea. Even completely well intentioned, it just doesn't work out well. And the more diverse a society, the worse it's going to work.

[knollimar]

How do you combat review bombing or someone just being bigoted?

[bell-cot]

The same ways that you have to combat fudged scientific data and didn't-write-it authors and lazy reviewers and hate from people who are in the opposite school on the whole Alzheimer's Amyloid issue and shite-quality citatations and etc. etc. etc.?

Yes, there can be obvious patterns to money/friends/skin color/gender that should be watched out for. But if you're paying much attention to those, then your science will quickly become a mere side gig to your new Cultural Warrior job.

[bob001]

> social scoring and consequences and reputations rarely actually stem from one’s actions

In my experience, this viewpoint is most strong in people whose actions are the most direct cause of their social issues. Almost always these people strongly refuse to admit that their actions should or are under their control. At which point others start to avoid them.

There are large scale social stratifications due to various things however individuals live in a very wide band in my experience. Some make choices that put them at the top of that band and others make choices that put them at the bottom.

Let's take two of your examples:

> how much money someone has put in someone’s pocket

> who someone knows

Careers choices directly lead to having higher income and more money. Socializing and putting effort into building a social network gives you more people to know. Both are to a decent degree under someone's control and based on their actions. You claiming otherwise is exactly proving my point on people wanting an excuse for their poor historical actions.

[bell-cot]

> The problem is that everyone imagines ...

I'd phrase it "most people want to imagine". Or will claim they want to - since not believing sounds depressing, or suggests that the person is an evildoer hoping to escape justice. (But unfortunately, people usually disagree about exactly what would be "just" or "fair". While wanting to imagine that they don't. Yes, the problem just went meta.)

> when the reality is ...

Try asking some really old folks about how much drama, inequity, and nastiness there often was in social settings where everyone was the same color and gender, nobody was notably wealthy, and nobody had any great connections. Or talk to an experienced junior high teacher. Or read some history. Humans are quite capable of dividing themselves into camps over any "differences" that they're able to perceive. Or invent, since dividing themselves into camps is often the unspoken objective.

[dwattttt]

For a more mathematically rigorous treatment, negative feedback is fundamental to Control Theory; it allows us to stabilise processes that would otherwise go out of control.

[Eddy_Viscosity2]

Even monkeys have society and they tend to only throw poo at those who have done wrong by that society. Monkeys have and try to maintain reputations.

[tough]

Is it reputation or hierarchical standing or just "power" within the society what makes them not throw poo tho

[Eddy_Viscosity2]

I'm not sure what you mean, but not throwing poo is the default behavior. They only do it when feeling stressed, or zoo settings they might even do it to get a reaction from the crowd. Figuratively speaking, no different than humans.

[csallen]

People react negatively to this, because they fear a dystopian society of the sort we've seen in plenty of movies, and rightfully so.

But it's also worth pointing out that "consequences and reputations stemming from one's actions" is already the world we live in and always have lived in. Hell, even Hacker News has karma points, downvoting, shadow banning, and the like. There's no such thing as a society with zero consequences and zero reputations. The only real question is a matter of degree, structure, severity, reach, and various idiosyncrasies that differ across cultures.

So it would be nice to have a more nuanced discussion about this instead of treating it like a 0 or 1 decision.

[unknown_user_84]

this reminds me of an episode of the TV show. the Orville that I recently watched where they stumbled upon a planet that had the social credit scoring system and everybody had little badges with literal upvote and down vote buttons on them. and when your grandma buys you ice cream and you're a little kid you say thank you and you hit the upvote button and she likes you even more because you hit the upvote button then because you said thank you.

and so it was obviously portrayed as kind of you know for the story and it worked and I like the episode and everything but it got me thinking in terms of a society like that of people could probably construct something that used that kind of a system but in a more regulated way and a more humane way and I think that's turns into the question of do we want to try and make this new thing or do we want to try and fix the old thing?

regardless, it got me thinking about some kind of a potential implementation of a formal social reputation system that essentially mirrors what goes on informally now, but makes it visible and makes the actions of affecting somebody's reputation visible.

My basic intuition here and for most things that deal with society or whatever is that Satan lives in the details and infinite potential dystopias and the potential for extreme inhumanity lurks in every crevice and corner. unknown unknowns and shit.

[bob001]

> regardless, it got me thinking about some kind of a potential implementation of a formal social reputation system that essentially mirrors what goes on informally now, but makes it visible and makes the actions of affecting somebody's reputation visible.

In general you can already get this by asking and taking feedback from others. I've seen a lot more people shoot or ignore the messenger in those cases then actually take the feedback. Everyone is the hero of their own story.

[nightshift1]

The Orville is excellent. I am a fan. Here is another show/episode that is very similar and relevant here : https://en.wikipedia.org/wiki/Nosedive_(Black_Mirror)

[cucumber3732842]

>Hell, even Hacker News has karma points, downvoting, shadow banning, and the like.

Internet Right/WrongThink scoring should be an argument against any sort of formalized or standardized social credit, not for it.

[globalnode]

and then you have to ask who defines good and bad behaviour and who determines consequences when power is lop sided.

[BobbyTables2]

Isn’t China doing something like this ?

Wonder if it could degrade to something worse than the status quo where reputations are falsely tarnished by competitors purely as a weapon to get ahead…

[nneonneo]

The "Chinese social credit" system is vastly overblown.

I'm living here (China, Beijing and Jiangsu) now, and there are exactly two cases in the past eight months where the "social credit" system has had any impact at all:

- To open "take first pay after" vending machines. These are vending machines which are basically big locked fridges; if your credit score is high enough, you can open the machines, take whatever drinks/snacks you want, and they'll use (I presume) computer vision to charge you afterwards. If you don't have a high enough credit score, tough, you can't use them. (But there's almost always normal pay-first vending machines nearby).

- To borrow mobile charging packs (powerbanks). Some operators will let you "swipe your credit" (check your credit score) to take one without paying a deposit. If your credit score isn't high enough, you first pay e.g. a ¥99 deposit (~$15USD) which gets returned when you return the powerbank, not a big deal.

That's...it. My credit score is high enough on only one platform (Alipay), so I get to try what happens with both "high" and "low" credit, and I can confidently say these are the *only* two cases where I have even been asked to show the credit score. I have taken multiple train trips, bought lots of stuff in stores and restaurants, etc. without ever touching the "social credit score".

P.S. I literally don't even know how to raise my credit score, and neither do many of the folks who live here - again, not that it matters, because the score is simply not that important.

[seanmcdirmid]

I haven’t seen normal pay first vending machines anywhere but airports. I’m in Chengdu ATM, but there is usually a family mart or Lawsons to go to anyways.

[deepsun]

But you're talking about Beijing and Jiangsu. If you go to Xinjiang, bad score can get you literally imprisoned and tortured.

[seanmcdirmid]

How many times have you been to Xinjiang? I’ve only been once, and didn’t notice any credits scoring going on even among Uighurs. I get that something bad must be going on sometimes, and maybe north Xinjiang (more populated, more Han) is a lot different from south Xinjiang (more Uighur), but if you visited Urumqi or Kashgar tomorrow, or even lived there, you probably wouldn’t notice much.

[AlexeyBelov]

It's funny how non-Chinese people explain to Chinese people how things work in China.

[rogerrogerr]

I remember in the west a few years ago we were seeing purported pictures of electronic billboards in China shaming low-social-credit-havers. Was that all a weird psyop?

[scns]

> electronic billboards in China shaming low-social-credit-havers

It is people who did not pay their debts. Dr Jonathan Tam has a video on YT explaining it called:

Why the World Fell for China's Fake Dystopia

https://www.youtube.com/watch?v=Ecx45eOuc0E

[inigyou]

But the USA has a social credit score that affects a lot more about your life, doesn't it? Like whether you're allowed to have a house or a car.

[ben_w]

Social credit? Or money credit?

(Genuine question, I'm not American and don't have any desire to move to the US).

[xoa]

>Social credit? Or money credit?

Neither. Above poster is wrong. You don't require anything to own a house or car in the US except your own cash/equiv. If you wish to borrow other people's money or property (in some cases at all when counterparty risk is high enough or buyers competitive enough, in most cases it's more a matter of interest rate), a sufficient credit score is the most standard and easy way to "make the case" that you will repay. But that's truly not at all required. I bought my current truck for $8100 cash on the table, I just showed up at the other guy's place and inspected it and chatted, looked good enough, paid him and we did the title change document, I screwed on a new plate and off I drove with it. Effectively zero dealers will not accept cash upfront. You can buy land and just build your own place.

Obviously, a lot of us find having credit very useful. And it's also both convenient and good for fairness/economic velocity/efficiency not to generally have to explicitly put up any extra collateral to get it or go through some complex old fashioned social dance. Hence "credit scores" that systemize a lot. But claiming they decide "whether you're allowed to have a house or a car" warps things badly.

[inigyou]

They are the same thing. The Chinese social credit is mostly about money. The USA money credit is affected by social factors. They are the same.

[gspr]

There's a huge difference between a completely decentralized reputation system like the interpersonal one that has existed since humans first became sentient, and an artificially built top-down reputation system run by a powerful entity (be it the state like in China or megacorps like in the West).

[attila-lendvai]

in short: our social sructure has built out so far beyond the Dunbar's number that it has become a haven for psychopaths.

[azan_]

People should get paid for reviewing. That’s the solution. Publishing companies rack in billions of dollars in pure profit exploring free labor. Once people actually get paid for reviewing, it becomes much faster and higher quality and you won’t need AI triage.

[Sharlin]

Doesn’t work anymore because people would just have ChatGPT generate their reviews and get paid for doing nothing.

[zx8080]

So in sum:

- it did not work before because greed

- will not work anymore because AI

Is human science dead?

[inigyou]

Science as an activity will never die, but science as an institution is in big trouble.

[NetMageSCW]

That should be will not work anymore because of greed.

[StableAlkyne]

Nah, people will just come up with different (probably broader) heuristics when reviewing.

We have a past example of this in the US, too. Certain countries have historically had a huge problem with paper mills. Because of this, most people in the US/EU do have a negative bias once they see the author affiliations of those countries. Yes, it isn't fair to researchers who aren't pulling some shenaniganry. But it is a common shortcut; the sixth or seventh time you've spent a few hours reviewing a paper filled with bullshit, you probably would develop it too.

I imagine we will see similar things come up: the most obvious I can think of is, if it's not a well-known institution then it might be bullshit.

Hell, we are already seeing something similar in FOSS, where many high profile projects have completely banned AI contributions due to all the low effort slop PRs.

[afdbcreid]

Then perhaps people need to pay to get their paper reviewed?

[azan_]

People already do.

[tollgategit]

Alternatively, put the pressure on the incoming requests for review.

Pay a deposit to have your work reviewed, if it's accepted as a good faith submission, the deposit is returned minus a minimal, irrelevant fee, if it is deemed in bad faith, take all of the deposit as a time-waste tax. With enough cases going wrong, the flood of slop slows down and maintainers have to pull the trigger as often.

This was the solution several people had proposed for Git issues, but none that I know of took the plunge. I really think curl should have done that.

[NetMageSCW]

Isn’t that an incentive for the AI under every rock people (who don’t believe real people use em-dashes) to see AI everywhere and get paid for it?

[ludicrousdispla]

The 'named' reviewers typically pass on the review responsibilty to grad students or less-senior colleagues. This is currently a generally accepted and sanctioned/encouraged practice. A social scoring system would need to eliminate that for true accountability.

[gradstudent]

Reviews are typically in conjunction with junior colleagues and the senior person is involved in the discussion and held responsible for the outcome. The junior folks then end up in the proceedings as external reviewers. In my experience anyway. I don't see an issue here. How else do you train junior researchers to review? Its how I learned and all those around me as well.

[weiliddat]

Yeah it's a well-known secret, and I think any sort of scoring system misses the point. The scentific method (and system, incl. peer reviews) is a public good (like good governance) that requires moral acknowledgement, the necessary social/cultural/political/academic pressures, and discipline of taking care of it by participating parties.

If you add a score/metric this incentivizes the wrong thing, like how money, funding, and career already incentivizes all the wrong non-scientific behaviors (like faking data, p-hacking, etc.)

[StableAlkyne]

> If you add a score/metric this incentivizes the wrong thing,

Don't we already have all of these things because academia decided to use H-index as a metric for career impact?

[weiliddat]

Yeah exactly. One more metric isn't going to solve it.

[whatever1]

Exactly. We invented a pump.

We can use it as vacuum to clean things up. The same pump can be used as a shit fire hose.

The effort we need to clean things up is significantly higher than the effort needed to make a mess.

[hellohello2]

I think you are correct, its so hard to judge the quality of research that we have been using publication record/count as a proxy. Ultimately, having papers being easy to write is good, provided we find a better way of judging quality.

[Rebuff5007]

I think the good solution is to introduce a bit of "shame" into the system -- de-anonymized submissions and reviews.

You'd be less willing to publish ai slop if your name had to be tied to it for years to come.