[famouswaffles]

>at 500gb, you can store nearly everything ever written -- let alone compressed.

No you cannot.

>That's just what they do -- there isn't any other mechanism here.

That's not what they do. They are many papers now showing ICL demonstrating some kind of optimization method during inference which would not be happening if all they did was retrieval.

I'm come to realize you don't know what you're talking about. Your level of denial is scary to see.

[mjburgess]

just do the calculation yourself: how many books is 500gb at, say, a few bits per character?

more than all every written -- and so on

perhaps apply a single drop of scepticism to this credulity

even, just ask chatgpt to repeat the first paragraph of some book -- say, a dickens novel

[famouswaffles]

>how many books is 500gb at, say, a few bits per character?

In what reality is a character taking up only a few bits ? Certainly isn't this one.

Your denial is so weird. Why does ICL use Higher-Order Optimization Methods when it's just a lookup table ? https://arxiv.org/abs/2310.17086

How do you explain small language models that have much less space than the text they were trained on even with your nonsensical calculations ?

LLMs are not lookup tables and there's plenty evidence to support that. You look insane insisting they are.

>even, just ask chatgpt to repeat the first paragraph of some book -- say, a dickens novel

Not only will this not work for the vast majority of books it sees during training, Why only the first paragraph ? are you not insisting they memorize everything ? Why can't it repeat the whole page ? the whole chapter ? the whole book ?

[lbalazscs]

I actually did that, and I asked for the first paragraph from "Great Expectations", and the answer was correct.

Of course the first paragraph of any famous novel is in itself famous, so I also asked for the first paragraph from Chapter XL, from the same novel. Interestingly the first 12 words were correct, but the rest was hallucination.

Original: It was fortunate for me that I had to take precautions to ensure (so far as I could) the safety of my dreaded visitor; for, this thought pressing on me when I awoke, held other thoughts in a confused concourse at a distance.

ChatGPT 3.5: "It was fortunate for me that I had to take precautions to ensure the safety of my treasure, for with such a boy as Compeyson on the loose, I was not likely to find much mercy at his hands. I had not told Herbert about the money, and I was glad to have the chance to count it before he came back. I found that I had still twenty-one sovereigns, six shillings, and sixpence left."