[neuronexmachina]

The LLM prompts are pretty interesting, e.g.: https://github.com/imelnyk/ArxivPapers/blob/main/gpt/utils.p...

> "You are an ArXiv paper audio paraphraser. Your primary goal is to rephrase the original paper content while preserving its overall meaning and structure, but simplifying along the way, and make it easier to understand. In the event that you encounter a mathematical expression, it is essential that you verbalize it in straightforward nonlatex terms, while remaining accurate, and in order to ensure that the reader can grasp the equation's meaning solely through your verbalization. Do not output any long latex expressions, summarize them in words."

[nicklecompte]

There's no way the prompt actually works, though. LLMs are not able to reliably "preserve the overall meaning" of things unless they're doing direct technical translation. The problem is going to be even worse with original research, because the LLM will try to summarize according to old ideas from blog posts / etc in its training data, and not the new ideas in the original research. In general document summarization is one of the worst use cases for LLMs, both in terms of its reliability and the difficulty of finding errors - how would you know without reading the paper? I would be surprised if this prompt worked on a single honest[1] paper that was written after the LLM was pretrained.

The bit about translating LaTeX expressions into human-comprehensible math sentences is interesting and AFAIK should work on something like GPT-4. But that's just a case of technical translation. GPT-4 definitely cannot "rephrase the overall paper... simplifying along the way." GPT-4 can't even summarize corporate reports without screwing up facts and figures - why on earth would you try to use it to summarize new scientific research?

Stuff like this is why I'm so concerned about LLMs: this prompt doesn't work, and people using AI for this stuff is just automating ignorance. Very frustrating.

[1] I say "honest" because this prompt would probably do ok on stuff coming out of a paper mill - the problem is carefully stated original ideas. GPT tears original ideas to shreds.

[bionhoward]

Can somebody please pay me a nickel every time someone states a belief LLMs cannot perform some task or another?

[Y_Y]

LLMs can't give you nickels

[bravetraveler]

Just put the words in there, signed The Prompt Engineer