Thank you for verbalising the issue, overconfidence was the risk I was seeing all along. Widely accepted, shamelessly overconfident, bus still potentially gibberish generator.
Thanks for calling these things out. I didn't see at first glance how language models could be disruptive in these fashions. I am doing research for an article on the decline of civilization. I think that this topic could be a partly related to this topic. Any resources you can point to will probably help all of us.
Unfortunately I don't really know of resources to point you to. These are mostly my thoughts. Most probably though influenced by a diet of sci-fi. I can however expand on them.
For somewhat recent and relevant content check The Orville S03E04. At the time it was just a critique of fake news but I think it is correct about what can happen if AI makes fake news easy to scale infinitely and with very low barriers to entry.
I think presently worrying about Sentient AI is misguided because Generative AI is equally serious and much more urgent. There is no need of further advancement. What I said is possible with current AIs already.
If I ever finish it, I can let you know about it somehow. I'm currently using the material for a science fiction story, but eventually I am going to want to write an article about what I learned. If you want to drop me your contact details, you can reach out to me on my hacker news profile and I will send you details if an when its finished.
I believe such a generator can be very harmful in our society that has not yet figured a way to scalably teach critical thinking.
Harmful at societal level by flooding the world with generated content therefore drowning actual information and by creating a lot of overhead.
Harmful at the personal level by increasing the difficulty of sorting information and by creating toxic dependencies (see the Replika scandal).
Harmful at the class level by further concentrating power towards plutocracy.
Harmful at the professional level by eliminating the lower levels of the professional ladder and therefore destroying the pipeline of skill.
And harmful at the interpersonal level by casting doubt on the authenticity of relationships.