[milemi]

I’m sure it outperforms the general population since most people can’t code a hello world, or regurgitate an answer to a problem they have been trained to answer but have no understanding of like ChatGPT can. But if a minimally competent human gave me a completely nonsensical answer to a question they haven’t seen before, the way ChatGPT does so confidently, I would think one of us had a stroke.

I would expect a journalist who never had do think through a theory of computation course to make breathless claims that ChatGPT can “solve programming problems“, but I’m pretty surprised to hear so many people who have jobs in tech repeating these claims, especially since all it would take them to trip up ChatGPT is a few seconds to type in a slightly unfamiliar or non-trivial question. It’s like a kind of second-order Turing test: if you think ChatGPT can program, you’re not a real programmer.