ChatGPT picks up 80% of the meaning and rewrites it in beautiful prose. Or maybe another language, in the style of Shakespeare.
On the other hand, if you're in a field where there's an adversarial use of text and the uncomprehended 20% might be used to nullify, contradict or make loopholes in the main body, then relying on ChatGPT is similar to using Tesla Full Self-Driving in a construction zone, near firetrucks, during a snowstorm.
Every summmarization is a choice of salience: what to include and what to leave ou, and how to express something in a different way.
The failure foolishly and misleadingly called “hallucination” is only one manifestation of an attribution error. If your summarizer leaves out something very important because it doesn’t understand it the result will be quite misleading.
For your average web text which these days is 90% filler and not important anyway, this is no big deal. This particular lawsuit appears the same. But for anything important, I wouldn’t trust it.
In my experience it’s generally accurate when summarizing content provided in the prompt context. Where it can run into trouble is “recalling” (if you can call it that) content that it was trained on.
On the other hand, if you're in a field where there's an adversarial use of text and the uncomprehended 20% might be used to nullify, contradict or make loopholes in the main body, then relying on ChatGPT is similar to using Tesla Full Self-Driving in a construction zone, near firetrucks, during a snowstorm.