[conartist6]

LOL. So basically the correct sequence of events is: 1. The scientist does the work, putting their own biases and shortcomings into it 2. The reviewer runs AI, generating something that looks plausibly like review of the work but represents the view of a sociopath without integrity, morals, logic, or any consequences for making shit up instead of finding out. 3. The scientist works to determine how much of the review was AI, then acts as the true reviewer for their own work.

[Herring]

Don't kid yourself, all those steps have AI heavily involved in them.

And that's not necessarily a bad thing. If I set up RAG correctly, then tell the AI to generate K samples, then spend time to pick out the best one, that's still significant human input, and likely very good output too. It's just invisible what the human did.

And as models get better, the necessary K will become smaller....

[conartist6]

That's a strategy for producing maximally convincing BS content, but the scientific method was absent. I'm sure it was an oversight... ; )

[Herring]

That’s on you. You get to decide what “best” means when picking among the K, so you only get bs if you want bs.

I occasionally get people telling me AI is unreliable, and I tell them the same thing: the tech is nearly infinitely flexible (computing over the space of ideas!), so that says a lot more about how they’re using it.

[conartist6]

So to really think the person writing the paper is the right one to review it?