[magusdei]

I doubt it. Mitsuku, a purely rule-based chatbot, was already able to correctly answer almost all questions of this form in 2014 simply by querying a large knowledge base of common-sense facts.[1] On the neural net side, Google's seq2seq was able to answer questions like this around ~2016-2017, although I have no idea about the accuracy.

It would be more remarkable if GPT-3 couldn't solve these types of questions. It might be another problem with the prompt design.

[1] Incidentally, the article is wrong in claiming that the state of the art before modern neural nets was Eliza. Rule-based chatbots got quite advanced in 2013-2016, although they admittedly were never capable of the sort of "true" understanding and long-term coherence that GPT-3 seems to display.