This asking GPT for inference needs to stop immediately before it becomes accepted and commonplace. This is almost the same as "I flipped a coin and the gods spoke to me"
It's far worse, because people who should know better, people who should understand at least the very basics of how the underlying algorithm works, are deluding themselves into believing it's some kind of magical oracle.
Tangentially, I think there is essentially already the beginnings of a cult of AI, based around the promise of technological salvation from worldly toil with somewhat eschatological undertones.
Just waiting for "Show HN: Repent, sell your house and car, invest everything into eternal-salvation.ai before it's too late and you're left in the dust as we ascend to the stars!"