[gambiting]

>>Do they even have direct access to published works to use as reference material?

I mean, clearly, given that it did answer my question eventually. Also wasn't it a whole thing that these models got trained on entire book libraries(without necessarily paying for that).

>>I wouldn't expect any LLM to be able to respect such a request

Why though? They seem to know everything about everything, why not this specifically. You can ask it to tell you the plot of pretty much any book/film/game made in the last 100 years and it will tell you. Maybe asking about specific chapters was too much, but Neuromancer exists in free copies all over the internet and it's been discussed to death, if it was a book that came out last year then ok, fair enough, but LLMs had 40 years of discussions about Neuromancer to train on.

But besides, regardless of everything else - if I say "don't spoil the rest of the book" and your response includes "in the last chapter character X dies" then you just failed at basic comprehension? Whether an LLM has any knowledge of the book or not, whether that is even true or not, that should be an unacceptable outcome.

[calf]

Why though? They seem to know everything about everything, why not this specifically.

The problem with this line of reasoning is that it is unscientific. "They seem to" is not good enough for an operational understanding of how LLMs work. The whole point of training is to forget details in order to form general capability, so it is not surprising if they forget things about books if the system deemed other properties as more important to remember.

[gambiting]

>> if they forget things about books if the system deemed other properties as more important to remember.

I will repeat for the 3rd time that it's not a problem with the system forgetting the details, quite the opposite.

>>The problem with this line of reasoning is that it is unscientific.

How do you scientifically figure out if the LLM knows something before actually asking the question, in case of a publicly accessible model like Gemini?

Just to be clear - I would be about 1000000x less upset if it just said "I don't know" or "I can't do that". But these models are fundamentally incapable of realizing their own limits, but that alone is forgivable - them literally ignoring instructions is not.

[staticman2]

I wouldn't expect an AI to know exactly what happens in every chapter of a book.

Knowing the plot of Neuromancer isn't the same as being able to recite a chapter by chapter summary.

I tried this Neuromancer query a few times and results greatly vary with each regeneration but "do not include spoilers" seems to make Gemuni give more spoilers, not less.

[gambiting]

>>I wouldn't expect an AI to know exactly what happens in every chapter of a book.

Cool. I did - and turns out it can do it, just not without giving me some spoilers first.

>>vary with each regeneration but "do not include spoilers" seems to make Gemuni give more spoilers, not less.

I'm glad I'm not the only one experiencing this then.

[staticman2]

> and turns out it can do it

Not really- if you had examined the output closely you probably would have seen noticed it conflated chapter 13 and 14 or 14 and 15. Or you got very lucky on a generation. It definitely doesn't exactly know what happens in each chapter unless it has a reference to check.