[andybak]

> but it lacked the ability to step back and recognise it had taken a nonsense path.

Errr. Yes! We're at the stage where you should be amazed how capable these things are considering their fairly brute force nature. Being surprised at the limitations is a peculiarly - dare I say? - human response?

Usually the workflow involves:

1. Write a naive prompt

2. Gain some insight into AI's behaviour

3. Improve prompt and repeat 1.

Of course they lack insight. Of course there's no "meta-reasoning". What is astonishing is how often a glorified Markov Chain gives the impression that there is both.

[vertis]

It's all astonishing. I don't think it's so much surprised at the limitation as trying to explore the edges. Enjoying finding some of the emergent behaviours.