[qingcharles]

Right, I know what you mean. If the parties are only breezing over the motion then it looks great and 95% of the time you'll get away with it, even though really it's ethically dubious. And that's a super hard one for a human to catch when reviewing LLM output. Especially because (certainly for me) you tend to get lazier and lazier reviewing the LLM output as they get "smarter."

I'm assuming you've just used some off-the-shelf ones like Claude or GPT? All the lawyers I know are just using those. I'd love to know what Lexis and Westlaw and other companies are serving that might mitigate some of these issues with better custom tuning or a better harness.

[lawtalkinghuman]

I have tested Lexis AI once for a legal research point. I wasn't particularly keen on putting the exact details of an actual problem in, but I gave it a summary version.

It didn't feel drastically different from using ChatGPT with the ability to search the web, except it was searching material on Lexis, both statute/case law and commentary. It dug out some commentary that confirmed my prior hunches, but also pointed to some cases that weren't in any way relevant.

Otherwise, all the experimentation I've done is with non-confidential material using public LLMs.

[qingcharles]

OK, sounds like they've basically done nothing more than attach an MCP with all the case law. I was hoping all that cash they make from monetizing the commons would be used to at least create a decent legal LLM.

It'll take another company eating their lunch for them to wake up.