How would you do it, tree search? If yes, I tend to agree with your initial statement that one should be able to teach LLMs to apply simple heuristics before considering it AGI.
I don't know the answer to your question "how to build AGI". Although if I had to guess, the AGI will probably have a supervisor algorithm (trained by RL), which will issue internal commands to pattern matchers (like GPT-4), to drive them to solve the problem. The supervisor algorithm will only have a little tacit knowledge about any specific problem (like language or world facts), only tacit knowledge about learning and reasoning, and how to do it economically.
So the supervisor algorithm will do the tree search if needed.
Yes, but does it because it read somewhere? Can it adapt the existing solution to a new variation? Can it solve a similar problem with different things? This is what humans do all the time.
If I add a small condition that makes the solution impossible, will it recognize that? Will it recognize for your example that it's a variation? Will it still be able to solve it when it is just a subtask of a bigger input?
If I ask it a leading question that intentionally relies on a wrong solution, will it recognize that?
I can't answer your questions (yet) but I have an observation. In the left sidebar where it summarizes the conversations in its "own words" the canonical example gets tagged with "River Crossing Puzzle" while my variation with the chasm gets tagged as "Transporting animals across chasm".
In the canonical example it also remarks "This is a classic river crossing puzzle" before delivering the solution.
EDIT: I tried some variations with "two wolves and a rabbit" and "three wolves and a rabbit". ChatGPT started bullshitting about its solution that supposedly was "a bit more complex". It started with taking one wolf to the other side. After I pointed out the rabbit would be eaten by the remaining two wolves it apologized for the mistake but just kept going hallucinating "correct" solutions.
For example: You have a goat, a wolf, a cabbage and you want to cross a river...