[emmender2]

I was waiting for: "but humans do that too" and bingo.

on another note: an entire paper written on one prompt - is this the state of research these days ?

finally: a giant group of data-entry technicians are likely entering these exceptions into the training dataset at openai.

[gwbas1c]

> an entire paper written on one prompt - is this the state of research these days

Years ago I attended an AI meetup where the organizer would discuss the differences between reductionist and wholistic research.

This is reductionist research.

In this case, reducing the paper to one prompt is to allow for objective comparison of models.

Otherwise, without a reductionist approach, how does one perform such an experiment and ensure that they are comparing apples to apples?

[emmender2]

what I would like to see is a parameterized class of prompts which can never be solved by the LLMs even when a finite number of them are manually added to the dataset.

[gwbas1c]

Well, should we consider ChatGPT a Turing machine? Otherwise, I think an answer like that either requires significantly more research; insight; or general knowledge about how LLMs work.

IE, you're getting into areas that are analogous to Turing's theories. I don't think he came up with those theories overnight.