[Terr_]

The un-solved problem is how to ensure users actually verify the results, since human laziness is a powerful factor.

In the long run, perhaps the most dangerous aspect of LLM tech is how much better it is at faking a layer of metadata which humans automatically interpret as trustworthiness.

"It told me that cavemen hunted dinosaurs, but it said so in a very articulate and kind way, and I don't see why the machine would have a reason to lie about that."

[heycosmo]

I would like to see solutions (for professionals) that ditch the whole generative part altogether. If it's so good at finding references or identifying relevant passages in large corpora, just show the references. As you said, the "answer" only entices laziness and injects uncertainty.

[Terr_]

I think the important product-design issue here (which may be sabotaged by the Attract Investor Cash issue) is that labor-savings can backfire when:

1. It takes longer to verify/debug/fix specious results than to just do it manually.

2. Specious results were not reliably checked, leading to something exploding in a very bad way.

[gowld]

Yes. The most "exciting" part is the worst part of the whole system, that contributes negatively.

[anon373839]

Perhaps the system should be designed to equivocate on any conclusions, while prioritizing display of the source material. “Source X appears to state a rule requiring 2% shareholders to report abc, but I can’t say whether it applies: [Block quote Source X].”

[Terr_]

That would be nice, but I cynically suspect it's not something LLMs are constitutionally able to provide.

Since they don't actually model facts or contradictions, adding prompt-text like "provide alternatives" is in effect more like "add weight to future tokens and words that correlate to what happened in documents where someone was asked to provide alternatives."

So the linguistic forms of cautious equivocation are easy to evoke, but reliably getting the logical content might be impossible.

[anon373839]

I agree, it is unlikely we’ll be able to get LLMs to provide “informed uncertainty” because they can’t interrogate any internal confidence in the correctness of the output.

But I wonder if tuning the output to avoid definitive statements would be beneficial from a UX perspective.

[Terr_]

I think it would help curb people over-trusting the model, yeah.

Heck, imagine how terrible the opposite would be: "When answering, be totally confident and assertive about your conclusions."

[TwitBar]

Arguments will be formulated by AI with another AI attempting to poke holes. You get a government appointed AI if you cannot afford one. This will kick off an arms race between plaintiffs and defendants. Legal companies then build moats around their bespoke AIs and it all boils down to a judge/jury voting based on a generated slideshow presentation (hopefully avoiding a miscarriage of justice /s).