[jsenn]

Sure, and human brains aren’t databases either, but it’s sometimes reasonable to say that we “store” and “retrieve” knowledge. All models are wrong but some are useful.

The question I’m asking is, how is this working in an LLM? How exactly do their weights encode (seemingly) the entire bible such that they can recreate long passages verbatim from a prompt that likely doesn’t appear anywhere in the training data (e.g. some vague description of a particular passage).