[creativenolo]

It seems like your issue is that this is not human, and it does not do what a human would do. This is a high expectation and, possibly dangerous, assumption.

Shows how this tech is such a double edged sword for user value, expectations and assumptions.

[vertis]

No, I'm well versed in the limitation of the tech. The comparison to what a human can do is just an interesting comparison. It's a feat the GPT-4 model can manage in this context. GPT-4 is also exhibiting emergent behaviours that are not found on models with smaller numbers of parameters, it is these emergent behaviours that make it seem more 'human' in it's ability to produce content, even if the "reasoning" about it is all an illusion.

Even here there are limitations, which was my intent with mentioning the Econ exam.

Aside: It's also possible to get different answers by making the LLM simulate being different people (a student, or a professor).

It is very easy to slip into thinking about ChatGPT in terms of actually thinking, but then again our understanding of the human mind has led to theories that conscious decision may well be an illusion as well (if brain scans showing that the decision is made before the conscious mind is aware are anything to go by).

At the moment though, these are just tools, with varying degrees of usefulness.