[jerf]

"I think a lot of this is based on circular reasoning."

No. Actually it's based on information theory, and probably a better model of what interacting with an LLM would look like a year or five later than the one you are operating on.

Here's a little hint: It has total amnesia. LLMs by their nature scale only so far, and while they may scale larger than ChatGPT, they aren't going to be scaling for an entire lifetime of interaction. (That's going to take another AI technology.)

Ever interacted with someone with advanced dementia but otherwise functioning faculties for any period of time? (I suppose they could well make good therapists too.)