[ahofmann]

"It's still pretty obvious and still much worse than human prose"

You have no idea how many false positives and how many false negatives you have in your judgement. It is indeed impossible to differentiate between badly written human text and somewhat good written llm text.

[lelanthran]

> You have no idea how many false positives and how many false negatives you have in your judgement.

The default LLM style is pretty deterministic. "Not X, Not Y. Just Z", etc.

There are phrases and cadences which are very rare in human prose (<5%) but unusually common (+95% occurrences in LLM prose). It is not unreasonable to look at content which is 95% LLM tells and conclude that that an LLM authored it.

I have noted, IRL, that those people who read very little, and only read when they have to (work docs, etc) are literally unable to tell that a piece of prose sounds like an LLM even when it has about 12 occurrences of "Not X. Just Y" or "Not X, Not Y. Just Z" in as many paragraphs.

[Izkata]

It's really similar to special effects in tv/movies: Whenever there's something new it's hard to tell, but as we get used to it and start seeing the patterns it becomes easier to tell. The older the special effects are the more obvious it is. Quite often you can just kind of tell if something was special effects or practical effects without being sure why. And there are always some really well done ones that slip past everyone, or odd lighting that makes it look fake (a thread a day or two ago on here about a legal case involving a photograph, some people in the comments thought it was a painting).

[jswelker]

You've found the "holy grail" of LLM writing differentiation! Your opinion is in the "Goldilocks zone" between vacuous and paternalistic!

[SwellJoe]

I wanted to be sure I wasn't lying to myself. So, I built this: https://prose-or-con.com/

I can still detect AI-written prose with pretty high confidence (I average about 85% accuracy after a bunch of rounds while testing, though I'm rushing because I'm testing), but it has gotten harder. Opus 4.8, GLM 5.2, GPT 5.5, and DeepSeek v4 Pro, are able to fool me sometimes. Mistral writes so weird it almost trips me up, but then I go, "oh, that's just Mistral". Qwen sometimes accidentally switches to Chinese (filtered out of the game, just a thing I found in testing). I'm doubling the corpus, currently, with a focus on the frontier models that have been able to fool me more often, so it'll probably get a little harder. GPT 4o is disgusting, just horrible saccharine nonsense (and, I guess that's the model that's caused the most psychosis with its sycophancy).

And, I have to concede that the best models prose has gotten better, especially when they're prompted to write in a specific style, like literary fiction or an encyclopedic article about a specific topic. Still flowery and verbose, though, and never really getting to the point (the AI doesn't want anything, and it shows).