[saghm]

> it correctly identified it as an imitation of James Mickens

How likely is it that it might take into account that it knows for sure it's not anything from Mickens from the latest training data? I'd be curious if it correctly identified a new piece from him that comes out as from him before it gets trained on it.

[jerf]

I fed an unpublished draft of mine to an AI. I saw it searching the internet and prompted it with the fact it could stop searching, it was not published. From there it guessed that it was me on the spot, which I thought was kind of funny. Can't deny the meta-logic there.

It referred to me by my login name on the AI site rather than the name it would have used if it actually found my website, so I think it was more logic than an actual identification, but it had clearly corrupted the search enough to no longer be a valid test.

Which does make me wonder about the original article; if the AI has in context any sort of clue that the user is "Kelsey Piper" (a memory of their name, a username of kpiper or kelseyp, etc.), that will radically tip the balance in favor of the AI guessing that way just by the nature of LLMs. That is to say, it highly increases the odds of that guess even if it's wrong.

Even if that is the case, though, the general identifiability of writing remains true. It's been shown for a while with techniques a lot less powerful than a frontier LLM.

[cg505]

The author specifically discusses their efforts to avoid this sort of information leak which would obviously poison the result.

[furyofantares]

She says she used incognito mode, as well as the API, as well as having a friend use their account.

[suriya-ganesh]

This is unlikely. The way model distribution works is that the model retains a lossy representation of James Micken's writing. Very likely, it cannot repeat Micken's writing verbatim. Neither can it reason about the training cutoff in this manner.

It's a lossy representation

[sausagefeet]

I haven't been following it well but isn't part of the NYT lawsuit against OpenAI that it sometimes spits out NYT articles verbatim?

[Iulioh]

Study: Meta AI model can reproduce almost half of Harry Potter book

https://arstechnica.com/features/2025/06/study-metas-llama-3...

[kbelder]

"The study authors took 36 books and divided each of them into overlapping 100-token passages. Using the first 50 tokens as a prompt, they calculated the probability that the next 50 tokens would be identical to the original passage. They counted a passage as “memorized” if the model had a greater than 50 percent chance of reproducing it word for word."

So they fed "It takes a great deal of bravery to stand up to our " and the llm responded "enemies, but just as much to stand up to our friends".

They repeated that for every 100 tokens of the entire book. I think lots of fans could do just as well. It's pretty good evidence that the potter books were in the training corpus, but it's not quite what people think when they say an llm has 'memorized' something. It's not like getting even a few pages out of the model.

[mapt]

Genome analysis is also a lossy process that chops the data up into tiny bits, like a newspaper sent through a shredder. We then piece together matching sequences in a sort of puzzle. It's often a relatively inaccurate solution. Then we try to do that again with a different copy of the newspaper sent through a different shredder. And again. A genome might be comprised of 10x reads, 30x reads, 100x reads, with more replications representing higher confidence.

There might be ten million people who have quoted Harry Potter at some point in their blogs or forum posts. There are only so many words in the books.

[algoth1]

That issue is different, when web tools were added to gpt4o it would fetch the site, and basically copy paste the text into the answer body. So, you were able to read the content of the site without the site getting the ad impressions. Now the system prompts put a very tight word limit - 25? - on quotes from sites the model visits

[spacechild1]

See also GEMA vs. OpenAI.

[sigmoid10]

It is lossy, but it is still enough for verbatim recreations. All of Wikipedia is just 24GB of lossless compressed text and all of JK Rowling's work fits into a few MB. So these things would easily be storable verbatim in trillion parameter models. Reasoning about the training cutoff is also something that the newest models do pretty well, because you can teach them to do so after pre training using e.g. SFT. With tool use it can then even check actual current sources, which may happen without you even knowing in the normal chat apps unless you use a controlled API call.

[saghm]

I feel like you're making a logical leap here by assuming lossy and failure to reproduce in entirety implies inability to recognize. As a trivial example, I can take a sha256 hash of your comment here, lose the ability to reproduce it, but still have an extremely accurate ability to recognize whether some text is exactly your comment or not. Obviously hashing every substring would not be a particularly efficient strategy, but my point is that saying "it's lossy" isn't particularly compelling without other details.

[gspetr]

The example you've provided just adds noise.

sha256 is deterministic, LLMs are not, even at temperature set to 0.

[devmor]

Haven’t there been repeated experiments that show if you jailbreak most frontier models’ harnesses you can get them to output near verbatim copyrighted works?

I swear there was a whole court case about this in the last year.

[koiueo]

How do you know, how the model works? If there was an index of all Micken's writings, or even if the model searched the web before feeding the response to you, you wouldn't know by observing from the outside.

[suriya-ganesh]

i suppose a quick test would be getting the model to write down Micken's essay end to end.

if the original essay was stuffed within the prompt window. the result will be word accurate.

unless this is a model trained specifically on Micken's essay (which claude is not).

[saghm]

This seems like a classic case of doing it being proof that it can happen, but not doing it being insufficient proof that it's impossible. I don't think there's a "quick test" of whether there might be a more effective prompt that would cause it to reproduce more effectively.

[repparw]

Didn't we get this with Harry Potter back in like gpt3.5? I'm sure I saw some news about it, someone getting it to output a book's intro word by word, couple pages?

[electroglyph]

that's in the ideal scenario where it's only seen a single copy of it tho