I wonder how much knowledge can be decoupled from experience, if at all.
If I read thousands of books that explain the details of another civilization in another galaxy, very thoroughly and consistently, but it it just happens to be all made up - did I gain knowledge? More importantly, does what I have in my brain now flip from being fiction to being knowledge if that civilization flipped from not existing to existing? How so, if nothing in my brain, or how I live out the rest of my life, changes in the least, if not a single atom in this galaxy changes (let's ignore that gravity has infinite reach and all that, for the sake of argument)?
If yes, how? What in your definition of knowledge makes that possible?
> If I read thousands of books that explain the details of another civilization in another galaxy, very thoroughly and consistently, but it it just happens to be all made up - did I gain knowledge?
sounds a lot like math - made up entities that very thoroughly and consistently fit together.
It's an interesting analogy you're making because... this is the lived reality of a lot of people that are interested in fictional worldbuilding / stories. And it flips to being real in the film Galaxy Quest.
Besides "secret" knowledge like the know-how at jobs, there's things like unwritten social etiquette (especially as it varies from place to place) or interfacing with physical world – reading about chopping tomatoes is different from experience acquired by actually chopping tomatoes.
It isn't. I constantly have access to non-public information, like the life of my peers and corporate secrets. Is it useful or essential or even desirable for LLM products? Hardly not, but it exists.
Edit: for "not in the training data" yes, humans generally can't know what they can't know.
If I read thousands of books that explain the details of another civilization in another galaxy, very thoroughly and consistently, but it it just happens to be all made up - did I gain knowledge? More importantly, does what I have in my brain now flip from being fiction to being knowledge if that civilization flipped from not existing to existing? How so, if nothing in my brain, or how I live out the rest of my life, changes in the least, if not a single atom in this galaxy changes (let's ignore that gravity has infinite reach and all that, for the sake of argument)?
If yes, how? What in your definition of knowledge makes that possible?