[quacksilver]

Could AI still be a useful tool if the reviewer performs a manual review first and then queries the LLM with:

1) Here is a new academic paper. Point out any inconsistencies, gaps or flaws in the research, and any contradictions with previous research in the field.

2) Here is a new academic paper and a journal submission policy. Does the paper meet the journal submission policy?

3) Here is a new academic paper, the review policy of the journal and a review of the paper. Does the review appear to have been conducted correctly.

4) Here is a new academic paper and a review of it. Has the review missed anything?

With the above, the reviewer could review the paper themselves, and then get the AI agent to proof read or double check everything, treating it like an editor / reviewer / secretary / grad student that they had asked to read the material. As long as the AI output was treated as potentially flawed feedback or a prompt from a third party to look deeper into something then that seems fine...

I'm surprised we are still using in-band signalling after the captain crunch whistle / blue-boxes have been around for that long

[rngrngrng]

No it cannot.

you are not allowed to share the unpublished results with anyone or any LLM, period. This is literally in every review policy (e.g. https://neurips.cc/Conferences/2025/CallForPapers)

[quacksilver]

Maybe I read it differently from you, but it states

"You can use resources (e.g. publications on Google Scholar, Wikipedia articles, interactions with LLMs and/or human experts without sharing the paper submissions) to enhance your understanding of certain concepts and to check the grammaticality and phrasing of your written review. Please exercise caution in these cases so you do not accidentally leak confidential information in the process."

From my reading then that would prohibit putting the paper into an openAI service, but how an interaction with a local LLM that doesn't involve sharing anything is treated is unclear. If you had an airgapped GPU rig running a local model and you formatted all storage on it after you were done, then no information would be shared, as you are just doing a bunch of math operations on it on your own machine.