[UncleOxidant]

> done by my clankers or clankers of other people

I'm getting so many requests to review LLM-generated documents - planning docs, docs intended for end-users, project docs, business plan docs. A team member sent me a zip file with about 30 LLM generated documents in it the other day and asked if I could review them right away. And a lot of it was just repetition and/or stuff that was just out of left field, made-up, hallucinated stuff. They're able to generate this stuff way faster than we can review. It used to be that it would take a significant part of a day for a project manager to come up with a planning doc - now they can generate one in a few minutes and send it out for review. It's just really tiring.

[mathw]

The only way to even start to counter that is to make it a firm company policy that if you use an LLM to hallucinate any documents you absolutely must thoroughly review them yourself before you send them to anybody else, and that you are still responsible for the quality of LLM-generated content.

Getting an LLM to vomit out a bunch of documents and sending them straight to another colleague is absolutely unacceptable behaviour.

[theideaofcoffee]

This is going to just run up against the insanity that is tokenmaxxing every moment of the day. When people are incentivized (upon pain of firing) to get the LLM to vomit out as much as possible, they're hardly going to stop and ponder if schlepping the slop over the wall is acceptable if the alternative is a pink slip.

Which is going to win?

[thinkingemote]

We employees need to remember that most software projects fail. So the work we produce should have lower value than we give it.

We also need to be motivated to stay in our jobs.

Most developers like their projects and value their work. But the chances are that it's for nothing.

Many developers know they work on bad products (gambling industry, military, surveillance, whatever) and so it's here that they focus on their technologies, tools and frameworks rather than the work they produce.

"Agentic engineering" for example.

Id be curious to see what and how Googlers are doing with their 20% time.

[soulofmischief]

The 20% policy at Google is effectively dead.

[AnimalMuppet]

If tokenmaxxing wins in your company, your company is going to lose. There's an external reality out there, outside your company, and your company has to produce things that actually work out there. Hallucinated AI slop does not help you do that. It leads you to unworkable plans, and if the plans produce, they produce unsellable products.

If you're an employee in that situation, push back if you can. If you can't, put your resume on the street. (But that may not work, these days. If it doesn't, all I can say is ride it out as best you can, and try to maintain both your job and your sanity. How? I don't know.)

[jaggederest]

Feed it into an AI and ask it to adversarially criticize it, doc for doc, send back 30 responses in a zip folder, wipe hands on pants, return to HN.

[nswango]

Don't know if you are serious, but why become part of the problem?

Why not just review a single document quickly, find an error which invalidates the document, and send it back saying "Policy paper 1 mentions X as being on the business plan for Y, it's not on the plan, please can you fix."

[lesostep]

Because they will fix it and send again.

Unless you can write a good-sounding reason why it's on them to review a LLM output before sending it to you, they will outsources this reviewing to you, and it's a lot of reviewing.

[nswango]

It's not a lot of reviewing if you simply find the first thing that makes the document unusable and call them out on it.

If it's genuinely hard to find that single bug .. perhaps the document has reached the quality required for corporate communication?

[lesostep]

Original comment stated that it was 10 documents, all LLM-generated.

In my experience, it does take a lot time and effort to find contradictions between 10 documents. Even with good documentation, it's hard to build a mental map for that amount of information.

[zufallsheld]

Because they'll just paste your remarks in their llm, let it correct the text and send it back to you.

[reed1234]

Why do your colleagues work when they could at least attempt it first?

[da_grift_shift]

This may actually be a solid way to tackle the bullshit asymmetry problem caused by drive-by LLM sloppers.

[jaggederest]

I always laugh because I've been practicing prompting every single day for the last few years, if they want to start a prompt fight, brother let me at 'em.

[sdesol]

> now they can generate one in a few minutes and send it out for review.

I think we will very soon move to a prove to me you've read it protocol and/or introduce speed bumps to slow things down.

[PsylentKnight]

Thats gonna be a no from me dog. I don’t expect anyone to read something I didn’t read myself

[intended]

Rate of generation/Rate of verification is a proxy for signal to noise ratios, just for work.

That ratio has changed, and verification is the hard part.

Verification is the point of all markets (and a decent part of human civ as well).

And review isn’t cost less - https://en.wikipedia.org/wiki/Ironies_of_Automation

[layer8]

> Rate of generation/Rate of verification is a proxy for signal to noise ratios

Hopefully you mean Rate of verification/Rate of generation.

[intended]

Yes! it should be:

Verification/generation

[soonnow]

Yeah I heard a similar thing recently at a presentation. At that point wouldn't it be easier to just send the prompt around?

[senectus1]

I find myself telling co-workers that if they couldn't be bothered writeing the email/document, why should i have to be bothered reading the document/email.

this is just spam, people sending unsolcited data at you and expecting you to swoollow and process that data.

its just rude and unreasonable, not to mention an unconscious (hopefully) act of sabotage.

[AgentMatt]

Seems like for such requests it's necessary to get some proof of work: require a meeting where for every artifact they sent you to review, they briefly explain the gist and point out the motivation for creating the artifact.

[AnimalMuppet]

Side benefit: They get public humiliation for the problems in what they sent around. It could create some social pressure to not send out garbage.

[jjav]

> I'm getting so many requests to review LLM-generated documents

That's the other nightmare of AI slop. So easy to generate endless content. Who will review?

Just today the boss request I review slides for a presentation. But it's all AI slop, generated from querying tickets and docs and who knows what. It's mostly sort of correct but also plenty misleading and incorrect. So now I have to fact check all this slop which will take hours (even with my AI assistance) and rewrite most of it.

If AI didn't exist, he would've had to do the research to generate the content and it would be 99% correct and I could just give a few notes of feedback in 5 minutes. But with the asymmetric AI workload, he can generate it in 5 minutes and I get to spend 3 hours correcting.

[AnimalMuppet]

> If AI didn't exist, he would've had to do the research to generate the content and it would be 99% correct and I could just give a few notes of feedback in 5 minutes. But with the asymmetric AI workload, he can generate it in 5 minutes and I get to spend 3 hours correcting.

Maybe, depending on the boss. Some would have spent five minutes describing what they wanted, and someone else would have spent three hours creating the deck.

[jjav]

I'm fine with that. The company doesn't have infinite people so as long as someone spends 3 hours generating and I spend 5-15 minutes reviewing, that's fine.

Problem with AI is that generation is so many orders of magnitude faster than reviewing so it's basically infinite monkeys on infinite typewriters.

[palmotea]

> A team member sent me a zip file with about 30 LLM generated documents in it the other day and asked if I could review them right away. And a lot of it was just repetition and/or stuff that was just out of left field, made-up, hallucinated stuff

You just discovered the unlock to massive AI-driven productivity increases: outsource the hard stuff to others, or just don't do it at all. Keep the easy tasks that generate a big volume of output for yourself.

[chadgpt3]

It's spam, it's a DoS attack. The right way to handle a DoS attack is to blacklist the sender. But this doesn't work if the sender is paying you.

[groby_b]

Write an LLM script to review them. Tell it to find at least three severe issues. Set to auto-reply.

He who brings the slop cannon shall be drowned by slop rain.

[tanseydavid]

> He who brings the slop cannon shall be drowned by slop rain.

If this approach gets widely adopted, then I think you should probably start building an ARK.

Make it big enough to hold two of every animal species. /s

[deterministic]

Reject documents that contain LLM hallucinations.

Or use LLM's to generate 12+ pages of detailed reviews of those documents and return to sender.

[freefaler]

I got this problem with my own employees, LLM are fine, but lazy slop is not permitted. Current idea is to have a clear "best practice" template for most of the research/specs/problem definition they submit and it reduced the slop to a manageable level. But this might work in a smaller company where the management is reading and is strict about these things.

[aakresearch]

Wait, what? I thought everyone agrees that modern models post September 2025 (or whenever Opus or whatever 5.6789 was released) do not hallucinate, make things up, contradict themselves and can review their own output into perfection regardless of task, goal or context???? /s

[UncleOxidant]

In general I think from the coding side they're more robust now. However, people generating docs are maybe not as experienced with how to prompt in ways that avoid having the LLM tell you what you want to hear. I think this is still a pitfall that can easily be fallen into. Those of us who are doing LLM-assisted coding for the last couple of years are more aware of this now. Those who are planning/management folks are still kind of susceptible depending on how much experience they've had dealing with LLMs.

[KronisLV]

What a take with no nuance.

> do not hallucinate

They do, just less. To the degree of being usable, as long as there are guardrails and they're used responsibly. For example, if there's code being output, there should be type checking and compilation, as well as code tests that prove that it works or that it doesn't - seeing how abysmal code coverage is in most of the projects I've seem, for whatever reason people thought that they didn't really need it much. They were wrong.

This also implies you need SOTA models on max reasoning.

> make things up

Same as above. Ideally you'd give them some way to verify their claims, like web search or browsing and referencing docs, Jira tickets etc., basically improve the signal to noise ratio.

> contradict themselves

They do so way less than before, as long as the above is true.

> can review their own output into perfection

They are pretty good at reviewing things, especially if you make them do adversarial review! It will never be perfect, but can be close in quality to human output (e.g. the code they produce, when used properly and with intent, is better than the code I've seen many developers write and ship before LLMs were a thing).

This also more or less scales with how much compute you give them - three parallel review agents will turn one output artifact into something good with higher confidence than two, and definitely better than with no review. There's a cost vs quality balance and it seems that all those xhigh and max reasoning modes are still geared way too much towards cost, instead of quality. So you have to make up for that shortcoming yourself.

> regardless of task, goal or context????

Garbage in, garbage out. I won't be an asshole and say that you're holding it wrong, nor will I say that anyone should listen to the claims marketing AI (absolutely delusional takes, meant to attract investors), but we're slowly getting to a better position in regards to LLMs, year by year.

It's just a shame that the peak of inflated expectations hit while the technology still hasn't fully plateaued and reached whatever its ceiling is.

I probably also shouldn't ignore the fact that some people will not care about any of it and send AI generated slop verbatim and to an outside observer there's no way to easily tell apart the difference between the two, unless you make a technical report contain exact references to where the data is sourced from, for example (and then either verify the references yourself, or make another agent do it).

[MaKey]

I think you missed the /s (for sarcasm) at the end.

[KronisLV]

Yeah, my bad, though I’ve also heard those arguments more or less said genuinely - on one hand people hold LLMs to some unreasonably high standard, expecting to one shot apps before being deemed good, and on the other just outputting slop with no regard for the quality.