[wisty]

Counter argument - even stone age Chat GPT 3 was great at making reasonably convincing sounding arguments and newer models are great at aligning those arguments to souces (laws or cases). Chat is IMO better at ambiguous nonsense than objective analysis.

[probably_wrong]

You may want to take a look at the "AI hallucination Cases" database which tracks cases when someone used AI and ended up presenting made-up sources in a legal case.

https://www.damiencharlotin.com/hallucinations/

Just because GPT can make an argument sound convincing it doesn't mean that the argument is convincing. Or based on objective truth, even.