[embedding-shape]

> [...] he had intervened at forces that were deploying commercially available AI tools before they had been properly assessed [...] “All forces have got a good policy on the use of Copilot,” Murray said. “All forces will have a policy that says, ‘Check everything that it produces’.”

Not only are they using AI before they've properly assessed them, they also end up using Copilot which must be one of the worse AIs currently available, probably because of existing Microsoft relations. And on top of all that, they hope to be able to rely on "Please review the outputs" which obviously isn't an actual solution here, of course people will get complacent and throw stuff over the wall whenever they can.

[Aurornis]

> “All forces will have a policy that says, ‘Check everything that it produces’.”

Everyone I talk to (including outside of tech) is going through this phase at their companies. It’s not working.

Checking the output seems like a simple request, but the question becomes: Check against what? If the police are making a document that sources from another report that another officer used AI to produce from their notes which were also run through AI and on and on, an inconsistency that leaks in at a previous step will check out when someone reviews the output against the inputs.

We’re all also discovering that many people’s idea of reviewing the output is to skim it and verify that it looks convincing enough. Checking facts is hard and takes time. These people are using AI because they want to work less, not to give themselves extra work.

[prymitive]

One can ask, what is a practical difference between “Check everything that it produces” and “Do all the work yourself”?

It’s not typing that’s the bottleneck, at least not often, so this is essentially assuming that you can do all the needed work without actually doing it, which is obviously wishful thinking.

[idopmstuff]

This is definitely the most interesting question in a ton of AI applications. I think folks should be really be spending a lot of time on figuring out how to deterministically check AI outputs in a way that's reliable in order to reduce the amount of work a human has to check, and to build tools that speed up the checking process.

Thinking about all of the fake citations in legal submissions that have come up of late, it seems pretty straightforward to set up a regex that captures all forms in which a cited case might be written (I could be wrong but I'd assume there's some standard variety of formats) and search those against a database (again assuming such a database exists) to ensure they all exist.

Then for the tougher problem of making sure that the cited cases say whatever the document citing them says they do, you could have an LLM run through the document, pull out the text with the case name and text about why it's being cited, then read the case and try to determine whether the reason for citing it is valid. Rather than just give a yes/no, you'd put the doc in front of the user and let them jump from citation to citation. On each citation, it'd pop up a card that shows the literal text of why it's being cited, a judgement from the LLM of whether it matches what the case says, and snippets of text from the case as evidence + deeplinks to that text within the case.

Or maybe you wouldn't even want to give the LLM's judgement since people might rely on that without reading, but there's definitely a way to speed up the review.

I believe OpenEvidence does something like this with medical papers. If you ask it a medical question, it doesn't answer so much as link you directly to the relevant papers so you can read them and determine if they're useful. Avoids all of the potential risks of using an LLM but still hugely valuable and time-saving for docs.

[iririririr]

excellent point. it is like saying computers in the 90s.

remember how the bank giving your money to the wrong person was a crime? and then when "the computet" did it was just business as usual and you paid more for banking because now they had "computer fraud" insurance?

same thing. cop deliver false report, jail (hah! i know). now, it was "the Ai". so no jail, they will go back and put rules for the cop to read or something.

and we are making everything worse by the minute. One gov push back on Ai nonsense, ibm/rh cames up with all sort of lies that would make any engineer or research laugh on their faces (federated learning being for privacy, instead of cost cutting. or explainable Ai being real, and not something bolted after the inference with extra unexplainable inference. etc.) but that are good enough to fool the regulator.

[ares623]

All this, so people like us can do a job that wasn't that hard in the first place (and in fact was quite comfortable all things considered), just a little bit easier, for companies that are promising to lay us off for productivity gains that aren't even measurable.

[iririririr]

my friend, jobs only matter in volume not quality.

why do you think every capital accumulation in the past caused "absentee landlords"? everyone wants the profit without even having to collect the rent.

Ai (or Bangladesh) doesn't have to do the work better than you, just deliver more X, where X is work output div capital input.

[lucaslazarus]

Depends, is P = NP?

[logifail]

> Checking the output seems like a simple request, but the question becomes: Check against what?

A colleague of mine circulated "minutes" from a meeting last week, there were only three of us in the meeting (one external service provider, my colleague + me).

There were several items on the "minutes" which I didn't recall being discussed, so I asked him if he'd had AI help, he said AI was filling in the gaps based on its knowledge of other discussions he'd had with it.

Glorious.

[WickyNilliams]

Even if you have some AI transcribe in real time, they introduce the kind of subtle mistake - like negating a statement - that is plausible, but undetectable in retrospect without the accompanying audio.

I often review the transcriptions that zoom produces out of curiosity, and this happens constantly. Makes them essentially useless as a statement of record

[simonw]

This is a great way of capturing the core problem. Fact-checking a document is a difficult skill! Expecting people who've never had to do that before to just start doing it - when these AI tools are supposed to save them time and make life easier - is not reasonable.

[jochembrouwer]

> Checking facts is hard and takes time.

This is a reasoning skill and I fear we are losing this skill if we do not update our education systems (all over the world) very fast. Children will use AI, and we should also see this as a tool. However, children now need to learn how to verify the output of the tool, and this is something what they do not do (and they also are not incentivized to do so). Which needs to change, otherwise the next generation of people exiting high school can use AI, but they have not learned how to reason themselves.

This also happens I guess in companies, AI is the simple way, the reasoning is the "hard" part which costs energy, and people need to be incentivized to do so. I have no idea how to create that incentive though.

[obscurette]

Education system has already lost and it happened before AI. The problem is that reasoning requires a very good literacy skills and these require a lot of practice to develop, but we stopped forcing students to practice this quite some time ago. A mass of illiterate students have been reality for several years all over the world.

[misnome]

> We’re all also discovering that many people’s idea of reviewing the output is to skim it and verify that it looks convincing enough.

I mean over time I've come to believe that most people are just _bad at reading_ - if you ask these people to compare two documents they'll say that they are the same if the wording or surface "feel" is at all similar, even if in the precision of the statement they say the opposite.

See also: People being generally bad at listening and hearing what they want to instead of anything quantitatively derivable from what the other person said.

None of this is written as an excuse for AI.

[_puk]

Here's a massive document, without any real context as to what thinking went into the points it's making, tell me if it looks ok. Oh, and there's 10 more where that came from.

We're outsourcing the thinking to the recipient.

Yes, it's way easier to create the report now, but it's not being honed down to the crux of the points it needs to make. And the reviewers are expected to what? Up their ability to mentally consume and reason about reports.

I mean, barrier number 1: did you read it yourself before asking someone else seems too high for some..

[_puk]

Never before has the adage "if I took* more time I would have written less" been true.

*Yes, I know

[throaway197512]

its just duplicating the work at that point, because you have to check everything anyways

[bluefirebrand]

> on top of all that, they hope to be able to rely on "Please review the outputs" which obviously isn't an actual solution here, of course people will get complacent and throw stuff over the wall whenever they can.

This is honestly the fundamental problem of AI as I see it

When we offload our work to a different person we can calibrate our expectations to our past experiences with that person. With AI the experience is not very consistent. To use AI effectively you basically should treat it as a low trust, brand new coworker every single time you use it

That doesn't really scale, so people have two choices: be constantly hyper vigilant for mistakes the AI makes, or become complacent and trust it more than they should

People rightly point out that humans make mistakes too, not just AI. But humans have a pretty manageable cap on the amount of output they can produce. One human can pretty thoroughly review the outputs of a small team of other humans

One human can't possibly thoroughly review the volume of output that an LLM they are prompting can produce

[gdulli]

Yeah, it's like declaring self driving safe because people are told to remain alert with their hands on the wheel, ready to take over in an instant. It's a charade.

[skydhash]

That’s why most people that says LLM doesn’t work. It’s not that it can’t produce a good output once in a while, it’s that you can’t guarantee it. Or reduce the risks of a bad output. It’s a chaotic element and the cost of being alert enough to ensure consistency (if it’s feasible at all) is higher than just doing without.

But AI proponents are more than happier to brandish carefully curated anecdotes than to do a systematic study of risks and impacts.

[analog31]

Something happening in the US right now is that the "presumption of regularity" is being openly challenged by judges. To the best of my understanding , it's the presumption that the testimony of the police is truthful until proven otherwise.

I think "check everything that it produces" will ultimately have to happen in cross examination on the witness stand. "Did you use AI" will be the first question.

[kerabatsos]

The mindset must be that if you use AI (which I happen to advocate for) you are also responsible for the output, if you use the output publicly. AI is obviously very powerful if used responsibly - the human is responsible for it once it is used - however it’s used.

[ajb]

I appreciate the sentiment, but this is exactly like the expectation that people can be responsible for intervening when self-driving or driver-assistance goes wrong. Human brains are strongly driven to conserve energy. If nothing seems to be happening - when errors become less and less frequent - the more difficult it becomes to guarantee intervention, and the less practiced the human will be at doing so.

I have written factory tests, in which I injected errors to make sure that the factory workers didn't develop "click next" syndrome and actually noticed errors. That's what you'd have to do. It's hard to get an organisation to stick to that, when they add up the time they're paying for in detecting fake errors.

[recursivecaveat]

Yeah, suppose a self-driving car is say twice as likely to crash as the average human (which could already be considered unfair since that includes like drunk humans and people who should've already lost their license). That means to monitor it's driving you will need to wait 25,000 hours until an accident occurs. That's ~ 12 straight years of a 9-5 spent sitting, doing absolutely nothing, just watching in a state of theoretical vigilance.

[techblueberry]

I think the problem is that, this is practically speaking impossible adjacent. I think generally speaking writing is way easier than editing, especially at scale. This isn’t binary or all or nothing, it’s not like “you can never use AI”. But I think we need to go back to augmentation over generation.

A person produces the content and AI removes barriers, and contextually accelerates the process keeping you in a flow state, rather than AI generates human edits.

[WickyNilliams]

Until case law establishes that people are in fact legally responsible for output, then it is just empty platitudes, I fear

[nyc_data_geek1]

If we need to check everything AI produces, every reference and factual assertion, where is the value in what it's producing? Why should we spend the money on producing this, and the time reviewing it, when I can just do it right the first time? Where is the efficiency?

[ggm]

This will self correct in a longer timescale as police prosecution fails to maintain a conviction rate and people ask why.

Hardly ideal but this isn't a static problem. Either the work gets done or cases get lost on slop.

[classified]

Not if solicitors and judges too offload to AI.

[netsharc]

Futurama being prescient: https://www.youtube.com/watch?v=Jqn8HB2w2Fs