[mostertoaster]

Sometimes I think the reason human memory in some sense is so amazing, is what we lack in storage capacity that machines have, we makeup for in our ability to create patterns that compress the amount of information stored dramatically, and then it is like we compress those patterns together with other patterns and are able to extract things from it. Like it is an incredibly lossy compression, but it gets the job done.

[ComputerGuru]

That’s not exactly true, there doesn’t seem to be an upper bound (that we can reach) on storage capacity in the brain [0]. Instead, the brain actually works to actively distill knowledge that doesn’t need to be memorized verbatim into its essential components in order to achieve exactly this “generalized intuition and understanding” to avoid overfitting.

[0]: https://www.scientificamerican.com/article/new-estimate-boos...

[halflings]

> That’s not exactly true [...] Instead, the brain actually works to actively distill knowledge that doesn’t need to be memorized verbatim into its essential components

...but that's exactly what OP said, no?

I remember attending an ML presentation where the speaker shared a quote I can't find anymore (speaking of memory and generalization :)), which said something like: "To learn is to forget"

If we memorized everything perfectly, we would not learn anything: instead of remembering the concept of a "chair", you would remember thousands of separate instances of things you've seen that have a certain combination of colors and shapes etc

It's the fact that we forget certain details (small differences between all these chairs) that makes us learn what a "chair" is.

Likewise, if you remembered every single word in a book, you would not understand its meaning; understanding its meaning = being able to "summarize" (compress) this long list of words into something more essential: storyline, characters, feelings, etc.

[cmpalmer52]

There’s a story by Jorge Luis Borges called “Funes the Memorious” about a man who remembers everything, but can’t generalize. There’s a line about him not knowing if a dog on the square glimpsed at noon from the side is the same dog as the one seen from the back at 12:01 or something like that. Swirls of smoke from a cigarette are memorized forever. He mostly sits in a dark room.

[DevKoala]

Thank you for reminding me of this story. He is my favorite author.

[mikewarot]

Long ago, I was introduced to the theory of Mappers and Packers[1], which are polar opposites in the ways that people can learn things. Mappers (like me) have a mental model of the universe which represents facts and knowledge as puzzle pieces that have to fit together into a coherent whole. Any inconsistencies in the fit between those pieces drive us nuts. When we encounter a new set of facts, we have a background process that tries to make them fit. Then all the new connections arise over time as we realize new ways we can combine old facts.

On the other extreme, are packers. They have optimized for packing facts in bulk, with little regard for how they fit together. If you give this type of person a set of instructions that require a wider knowledge of how things fit, they will get lost, frustrated, and/or need support. If you anticipate this, and spend a bit extra time to show how to handle all of the possible contingencies, (and give them a document of this) they're good, and will be quite happy with your support.

I think that mappers take more time figuring out the model, compressing the facts to save space, and increase applicability in general.

[1] https://wiki.c2.com/?MappersVsPackers

[JieJie]

My mind is a blurry jpeg of my life.

(https://www.newyorker.com/tech/annals-of-technology/chatgpt-...)

[ComputerGuru]

> but that's exactly what OP said, no?

Not precisely. We don’t know if verbatim capacity is limited (and it doesn’t seem to be) but the brain operates in a space-efficient manner all the same. So there isn’t necessarily a causative relationship between “memory capacity” and “means of storage”.

> Likewise, if you remembered every single word in a book, you would not understand its meaning

I understand your meaning but I want to clarify for the sake of the discussion that unlike with ML, the human brain can both memorize verbatim and understand the meaning because there is no mechanism for memorizing something but not processing it (i.e. purely storage). The first pass(es) are stripped to their essentials but subsequent passes provide the ability to memorize the same input.

[SanderNL]

We know for certain it is limited. Do brains not adhere to physics?

[astrange]

They only live 100 years and the write rate is only so high.

[whimsicalism]

> verbatim capacity is limited

I am but a simple physicist and I can already tell you it is.

[ComputerGuru]

I mean in terms of our ability to reach those limits, naturally.

[WanderPanda]

Compression = Intelligence

http://prize.hutter1.net/

[briansm]

Nope, more specifically:

Lossy compression = Intelligence

That's where the Hutter Prize falls down, it's based on lossless compression, which is nothing like how the brain works.

[jjk166]

Distilling knowledge is data compression.

[w10-1]

You're conflating memorization with generalization, no?

[jjk166]

Memorization is storing data. Generalization is developing the heuristics by which you compress stored data. To distill knowledge is to apply heuristics to lossily-compress a large amount of data to a much smaller amount of data from which you nevertheless can recover enough information to be useful in the future.

[staunton]

> Generalization is developing the heuristics by which you compress stored data

such that

> you nevertheless can recover enough information to be useful in the future.

I disagree (in case you meant to imply it) that compression implies generalization.

[jjk166]

I did not mean to imply compression implies generalization, if anything the reverse. Compression is the act of cutting, generalization is the whetstone by which you may sharpen a blade, which is the compression heuristic. A more general heuristic is to compression what a sharper blade is to cutting.

[nonameiguess]

I've thought about this a lot in the context of the desire people seem to have to try and achieve human immortality or at least indefinite lifespans. If SciAm is correct here and the upper bound is a quadrillion bytes, we may not be able to hit that given the bound on possible human experiences, but someone who lived long enough would eventually hit that. After a hundred million years or whatever the real number is of life, you'd either lose the ability to form new memories or you'd have to overwrite old ones to do so.

Aside from having to eventually experience the death of all stars and light and the decay of most of the universe's baryonic matter and then face an eternity of darkness with nothing to touch, it's yet another reason I don't think immortality (as opposed to just a very long lifespan) is actually desirable.

[mewpmewp2]

I imagine there would be perhaps tech or technique which you can choose to determine which memories to compress and countless of others techniques like extra storage that you can instantly access, so I don't see all of these as being real arguments why not become immortal. If I have to choose to be dead and memoryless compared to losing some of my memories, but being still alive, why should I choose being dead and memoryless?

And when losing memories you would first just discard some details, like you lose now anyway, but you would start compressing centuries into rough ideas of what happened, it's just the details that would lack a bit.

I don't see it being a problem at all. And if really something happens with the Universe, sure I can die then, but why would I want to die before?

I want to know what happens, what gets discovered, what happens with humanity, how far do we reach in terms of understanding of what is going on in this place. Why are we here. Imagine dying and not even knowing why you were here.

[p1necone]

My naive assumption would be that it would be a fairly gradual process. You'd just always have a sliding window of the last N years of memories, with the older ones being progressively more fuzzy and unreliable.

[imtringued]

Longtermists argue that we will be harvesting hawking radiation from blackholes trillions of years after the heat death of the universe.

[__loam]

The last civilizations will be built around black holes.

[oneTbrain23]

You obviously hand wave alzheimer and dementia. Human don't know exactly how brains works. The computational storage is just an estimate of what we understand von Neuman computer storing data 1 and 0. In every psychological test conducted on human mind, they clearly have a limit.

[downboots]

Can "distill knowledge" be made precise ?

[ComputerGuru]

As best as I’ve been able to research, it’s still under active exploration and there are hypotheses but no real answers. I believe research has basically been circling around the recent understanding that in addition to being part of how the brain is wired, it is also an active, deliberate (if unconscious) mechanism that takes place in the background and is run “at a higher priority” during sleep (sort of like an indexing daemon running at low priority during waking hours then getting the bulk of system resources devoted to it during idle).

There are also studies that show “data” in the brain isn’t stored read-only and the process of accessing that memory involves remapping the neurons (which is how fake memories are possible) - so my take is if you access a memory or datum sequentially start to finish each time the brain knows this is to be stored verbatim for as-is retrieval but if you access snapshots of it or actively seek to and replay a certain part while trying to relate that memory to a process or a new task, the brain rewires the neural pathways accusingly. Which implies that there us an unconscious part that takes place globally plus an active, modifying process where how we use a stored memory affects how it is stored and indexed (so data isn’t accessed by simple fields but rather by complex properties or getters, in programming parlance).

I guess the key difference from how machine learning works (and I believe an integral part of AGI, if it is even possible) is that inference is constant, even when you’re only “looking up” data and you don’t know the right answer (i.e. not training stage). The brain recognizes how the new query differs from queries it has been trained on and can modify its own records to take into account the new data. For example, let’s say you’re trying to classify animals into groups and you’ve “been trained” on a dataset that doesn’t include monotremes or marsupials. The first time you come across a platypus in the wild (with its mammaries but no nipples, warm-blooded but lays eggs, and a single duct for waste and reproduction) you wouldn’t just mistakenly classify it as a bird or mammal - you would actively trigger a (delayed/background) reclassification of all your existing inferences to account for this new phenomenon, even though you don’t know what the answer to the platypus classification question is.

[esafak]

https://en.wikipedia.org/wiki/Rate%E2%80%93distortion_theory

[clord]

imo, it amounts to revisiting concepts once more general principles are found — and needed. For instance, you learn the alphabet, and it's hard. the order is tricky. the sounds are tricky, etc. but eventually, it get distilled to a pattern. But you still have to start from A to remember what letter 6 is, until you encounter that problem many times, and then the brain creates a 6=F mapping. I think of it in economic terms: when the brain realizes it's cheaper to create a generalization, it does so on the fly, and that generalization takes over the task.

Somtimes it's almost like creating a specialist shard to take over the task. Driving is hard at first, with very high task overload, lots to pay attention to. With practice, it becomes a little automated part of yourself takes care of those tasks while your main general intelligence can do whatever it likes, even as the "driver" deals with seriously difficult tasks.

[pyinstallwoes]

It’s almost like a tuning fork. When the mapping becomes 6=F the two separate tuning forms are now sympathetic to a frequency.

[__loam]

Unless you know something the neuroscientists don't, it cannot.

[gattilorenz]

Is there a “realistic upper bound” in things that should be memorized verbatim? Ancient greeks probably memorized the Iliad and other poems (rhyming and metre might work as a substitute for data compression, in this case), and many medieval preachers apparently memorized the whole Bible…

[firecall]

Does the brain require more energy to store more information?

Or is it always running at the same pace regardless of if it’s empty or not?

I guess the Brian doesn’t really work like that…. But I’m curious :-)

[ReactiveJelly]

The brain doesn't seem to ever "clock down" when it's idle, which is interesting to me

https://en.wikipedia.org/wiki/Human_brain#Metabolism

> The energy consumption of the brain does not vary greatly over time

[gmadsen]

maybe between sleep and normal waking idle because there is actually quite a bit going on during sleep. There has been quite a bit of research though regarding higher "clock up" states consuming far more energy, such as grandmasters playing a chess tournament

[lobocinza]

My lazy research pointed to 6K calories which is a lot more than a typical person can burn by physical exercise.

[TheRealSteel]

You seem to have just re-stated what the other person said.

[whimsicalism]

Thank you, thought I was losing it for a second

[bufferoverflow]

There are rare people who remember everything

https://youtu.be/hpTCZ-hO6iI

[svachalek]

It's pretty fascinating to me how "normal" Marilu Henner seems to be. I'm getting older and my memory is not what it was, but when I was younger it was pretty extraordinary. I did really well in school and college but over time I've realized it was mostly due to being able to remember most things pretty effortlessly, over being truly "smart" in a classic sense.

But having so much of the past being so accessible is tough. There are lots of memories I'd rather not have, that are vivid and easily called up. And still, I think it's only a fraction of what her memory seems to be like.

[93po]

As someone on the other end of the spectrum, I have an awful memory, and don't remember most of my life aside from really wide, sweeping generalizations and maybe a couple hundred very specific memories. My way of existence is also very sad, and it makes me feel like I've not really lived.

[obscurette]

It's likely that you actually have memories about details, but don't have a way to recall these memories. I always wondered how the heck people write memories until I saw someone to do it. He used a lot of triggers – photos, newspapers, letters etc. Later I had a chance to visit museum where typical home environment of my childhood was exhibited (yes, I'm that old) and realized how many memories small things can trigger in my brain.

[93po]

I agree. There are definitely triggers for old memories and I have heard cannabis also adds flexibility to that recall

[TheRealSteel]

" I did really well in school and college but over time I've realized it was mostly due to being able to remember most things pretty effortlessly"

Same! They thought I was a genius in primary school but I ended up a loser adult with a dead end job. Turns out I just liked technology and was good at remembering facts and names for things.

[hgsgm]

Is there scientific evidence of that or just claims?

[bufferoverflow]

They were tested. They all correctly recall weather and major events on any day they were asked about.

[badumtsss]

some people don't want to be studied or tested.

[tbalsam]

For more information and the related math behind associative memories, please see Hopfield Neural Networks.

While the upper bound is technically "infinity", there is a tradeoff between the amount of concepts stored and the fundamental amount of information storable per concept, similar to how other tradeoff principles like the uncertainty principle, etc work.

[scrps]

Thank you

[mr_toad]

Artificial neural networks work a lot like compression algorithms in their ability to predict the future. The trained network is a compression algorithm - it does not store compressed data.

We don’t know if the animal brain works the same way, but I suspect it is mostly compression algorithms designed to predict things, and doesn’t store much data at all.

[bobboies]

Good example in my math and physics classes I found it really helpful to understand the general concepts, then instead of memorizing formulas could actually derive them from other known (perhaps easier-to-remember) facts.

Geometry is good for training in this way—and often very helpful for physics proofs too!

[lacrimacida]

Too bad this method is penalized most on tests (timed) where memorization is favored. But deriving results reinforce knowledge, understanding and patterns best in my opinion.

[pillefitz]

That is essentially what embeddings do

[nightski]

Maybe, except from my understanding an embedding vector tends to be much larger than the source token (due to the high dimensionality of the embedding space). So it's almost like a reverse compression in a way. That said I know vector DBs have much more efficient ways of storing those vector embedding.

[jncfhnb]

Tokens are not 1:1 with vectors.

[BSEdlMMldESB]

yes, when we do this to history, it becomes filled with conspiracies. but is merely a process to 'understand' history by projecting intentionalities.

this 'compression' is what 'understanding' something really entails; at first... but then there's more.

when knowledge becomes understood it enables perception (e.g. we perceive meaning in words once we learn to read).

when we get really good at this understanding-perception we may start to 'manipulate' the abstractions we 'perceive'. an example would be to 'understand a cube' and then being able to rotate it around so to predict what would happen without really needing the cube. but this is an overly simplistic example

[NovaDudely]

This was the thinking I was taking. It is a useful tool at first but taken too far can be a bad thing in some situations.

[pyinstallwoes]

Maxwell’s demon to entropy