To me, this is the quintessential risk: It's plausible enough it will fool somebody with authority to act, but lacking competency to understand the information is low grade. Boom! "oh man.. but the computer said it was ok"
Those attorneys who blindly used ChatGPT to generate a brief regarding MC99 liability (a field they had no experience in) are a good example, I think. Of course, in that case opposing counsel started looking at the cites and quickly had questions for them...
It's really incredible how plausible sounding ChatGPT's legal BS is. Completely hallucinated cases, arguments, citations (properly formatted for real reporters!), people, ideas... but if you just skim it, there wouldn't be any immediate way for a layman to tell it was total bullshit, and I'll bet an overworked attorney could be taken off guard, too. Obviously won't get you too far in actual litigation, and I really feel for the clients of any attorney that pulls such shit.