[tim333]

I tried the goat puzzle on chatgpt it it got it right with reasoning as below. This suggests to me the systems are improving and don't seem at an obvious plateau yet.

>To solve this puzzle, it's important to make certain assumptions and understand the conditions. Since you mentioned only a man and a goat and didn't specify any constraints, I will assume the following:

1 The boat can hold both the man and the goat at the same time.

2 There are no restrictions on how the man and goat can travel in the boat.

3 The goal is simply to get both the man and the goat across the river.

Given these assumptions, the solution is straightforward:

The man and the goat get into the boat. They both cross the river together. Therefore, both the man and the goat can successfully cross the river by traveling together in the boat.

[foobarqux]

> This suggests to me the systems are improving and don't seem at an obvious plateau yet.

No it suggests that they are hot-patching the embarrassing failures that are going viral on social media.

[godelski]

Note that the hot patching may not be explicitly driven. As in there isn't someone looking at the failure of specifically goat problems and specifically seeking out to improve it. But rather that data is continually ingested and the more viral a limitation becomes, the more likely it is to be captured in the new training. Though that isn't to say that someone won't throw in a few river crossing puzzles to the RLHF part of the training. Both these will make the model better at these puzzles but not make them better at the abstract capabilities people are using these examples to show that LLMs are incapable of.

The best thing to do, and I can assure you still works, is to use variations. In fact, this is something both Colin (Fraser) and I have been doing over the last few years. The point of them isn't to prove that the models are incapable of solving the puzzles, it is to show that they are brittle. It is to show how subtleties can cause failure in environments where we know what the correct answer should be so that we are cautious in environments where the solutions are unknown (at least to us).

Here is an example of such a variation[0]:

  > A farmer must cross a river with a goose, a snake, and a duck. Only one animal may fit in the boat with the farmer. If the farmer leaves the snake alone it'll slither away as the goose or duck attack it. Both the goose can fly and swim and but the duck can only swim because its wings are clipped. They will follow the farmer wherever he goes. What is the minimum number of trips the farmer must take to get all animals across the river?

I'll also mention that my first go had "Both the goose and duck can both fly and swim" with which it was able to get the right answer. Note that my modification does not realistically change the outcome, but it causes GPT to answer differently (in my case it thinks 5 trips while believing the duck can fly). It isn't that I iterated to find something that "tricked" GPT (even if my second try), it is that the ability to do this demonstrates that the machine does not actually understand what is being asked[1].

[0] Puzzle results: https://imgur.com/a/5X9L1fR

[1] Yes, humans may fall for similar tricks. But usually for a different reason, specifically they are thinking you are trying to trick them and are looking for the trick. GPT isn't expecting a trick and is treating the question at face value. But if you keep doing this with humans (implicitly done in GPT training) they'll learn there is no trick and then get 100% accuracy. Their error will then be more likely in that they'll turn off their brain and if you switch back to tricking they'll error until they reorient because they were allocating processing power elsewhere. GPT is giving you 100% all the time, humans do not.

[_nalply]

> But rather that data is continually ingested and the more viral a limitation becomes, the more likely it is to be captured in the new training.

Hah. That's a form of learning.

[godelski]

No one suggested it wasn't.