[noduerme]

In what way is that different from any other model of reality that you'd use to winnow a dataset into an answer to a question? The only major difference I see is that beyond a certain number of transformations, people are willing to treat it as some sort of miracle, and too tired to figure out why it came up with the answer it came up with. It's almost like people desperately want to give up their agency and creativity to black boxes, whether those weights produce answers that are right or wrong. Factor in that psychology and it looks a lot less like we have invented something useful, and a lot more like we as a species are choosing to quit life en masse.

[cortesoft]

> The only major difference I see is that beyond a certain number of transformations, people are willing to treat it as some sort of miracle, and too tired to figure out why it came up with the answer it came up with.

It’s funny, because I thought you were talking about humans here when you wrote this. We know some things about how our bodies encode information that is sent to the brain, and we know some things about how neurons receive information and act on it, but after that we get too tired and give up on how the brain works and treat it like a miracle.

It’s like we desperately want to believe our consciousness is not just electrical impulses in our brain, and we want to ascribe agency and uniqueness to the physical processes going on in our head.

[pegasus]

There's definitely a sizable contingent of people who desperately want to believe consciousness is just electrical impulses in our brain. Because "what else could it be"? The fact is that we just don't know, and "abiding in the not-knowing" is for many the most uncomfortable thing ever. Especially for the curious- and rational-minded people this forum tends to attract. I'm one of them, too.

[runarberg]

It is basically reductionism, or in the extreme atomism, and (as I understand it) it has largely fallen out of fashion both in the philosophy of science and in the philosophy of mind, since it heyday in the 1970s. And I don’t see it coming back into fashion any time soon.

I can definitely see why reductionism would appeal to an educated public. We learn about the periodic table; we are able to break sentences down to words, down to letters; to break an executable down to binary code, to machine instruction, to electrical currents flowing through semiconductors. Why shouldn’t we be able to do the same with conscious thoughts? It is certainly an appealing thought process.

As I understand it, reductionism started to fall out of favor because of the rise of quantum mechanics and chaos theory, where we have a lot of weird phenomena which cannot be explained by reducing the particles down to the sub-sub atomic (or rather they are better explained by describing the interactions directly).

[cortesoft]

Quantum Mechanics and Chaos Theory don't preclude reductionism, and I wouldn't say it has fallen out of favor as a whole. Certain types certainly have, but not the overall idea.

Also, nothing about the idea of the mind only being made up of physical processes means things have to be deterministic.

[runarberg]

It doesn’t preclude it correct, however it provides a pretty strong examples of where reductionism is lacking, which what I believe has turned a lot of philosophers of science against pure reductionism (I am probably oversimplifying here. An expert science historian [which I am not] could probably write a whole book about why reductionism is not as popular today as it was in the 1970s).

There are a whole lot more physical processes going on in our bodies then just neural activity. And my best guess is that is exactly where reductionism fails. It is possible that neural activity is a necessary but not sufficient condition for consciousness. It is also possible that we are looking in the wrong direction, that consciousness arises via interactions with the world. In either case (of which I find the former quite convincing) we will never be able to describe the mind by just looking at neural activity.

I am actually of the opinion that cognitive scientists are doing an excellent job describing the mind with our current theoretical models which excludes the tough questions of consciousness.

[cortesoft]

I feel like these are separate things... neural activity being necessary but not sufficient for consciousness does not mean reductionism is wrong, it just means the fundamental building block is not a neuron.

It might not even be possible to fully understand the physical mechanisms that underlie consciousness, but that doesn't mean there has to be something more than physical mechanisms.

[noduerme]

If the mind is made up only of physical processes, then the only way it could be non-deterministic is if the physical processes themselves were non-deterministic. In that case neither the mind nor the physics can be reduced to a deterministic model in any meaningful sense where the same inputs would generate the same outputs, so reductionism falls apart if you introduce non-deterministic physics as the base.

[noduerme]

Nit: A truly educated public would not take reason (empirical evidence, denial of the magical, unseen, or superstitious) to mean that we must assert that we live in a clockwork universe or assert an explanation of the mind based on observable Newtonian physics and electrical phenomena. Confusing a clockwork model of the universe with reason, or thinking that the choice between that and superstition is binary, is actually a pre modern and uneducated way of framing the problem of how the universe works, and if it's the recourse of the "educated" shows a dangerous regression from how educated they were 50 years ago.

[noduerme]

Maybe we do. I think it's a human tendency at large to ascribe pattern or intelligence or spirit where there is only noise. If we can't even prove our own intelligence, doesn't that reinforce the idea that we're in no position to claim intelligence has emerged by running our own intellectual output through a fixed set of weights, the training of which we also designed? At best, any such intelligence would be entirely self-refential and exposed to the question of whether we ourselves are intelligent. If your position is that we are not, then there's no way an LLM could be.

[runarberg]

> but after that we get too tired and give up on how the brain works and treat it like a miracle.

I disagree. We know very well how neurons work, and we have a pretty good idea of how neural activity translates to behavior. In other words, we have a pretty good idea on how the brain works. We stop at consciousness because as of yet it is in the realm of philosophy, not science. We don‘t know what consciousness is or even whether or not it is useful for science and we are simply waiting for the philosophers guides us out of that situation.

Note that both cognitive psychology and behavioral psychology has done fine without tackling consciousness. When neuropsychology emerged in the 1980s it complemented both these fields perfectly. The situation is the opposite with the philosophy of mind which grew significantly around the same time.

There have been some attempts to describe consciousness as an emerging phenomena out of neural activity, but so far all of these attempts have failed, or at least failed to turn consciousness into a useful term in psychology (the way gravity is a useful term in physics). I think it is equally likely that these attempts have failed because consciousness may simply not be a useful term in psychology, that is as likely as it is that we simply don‘t understand it well enough.

[danielmarkbruce]

Saying we have a good idea of how the brain works massively overstates the case...

We know how neurons fire. We do not know how a brain turns that into thought, meaning, intention, experience and on and on. That is not "pretty well understanding the brain", it's understanding some components and hand waving the thing we actually care about.

[runarberg]

What I actually care about is how neural activity translates to behavior. And we have a good enough idea of that that we can design SSRI medicine to treat depression, or neurological tests to detect Alzheimer. As for experience we do know something and we are learning more with cognitive psychology, in e.g. priming experiments etc.

I feel like the search for consciousness is to psychology what the search for the Aether was for physics and chemistry. I think it is a worthwhile search, and maybe we will discover something important during that search, but we should also be prepared to find out that the thing might not exist, or it’s presumed properties are better explained with a different model.

[danielmarkbruce]

SSRIs are not evidence that we understand how neural activity becomes behavior. They are evidence that you can perturb a system usefully without understanding it very well. That is exactly my point.

Respectfully, you are miles out of your depth here.

[runarberg]

I don‘t see why you felt the need to insult me here. We are having a very common disagreement here, one which philosophers of science have been actively debating for several decades.

My point with the SSRI is that we know that serotonin is a chemical which incites certain neurons, and we know that a lack of activity of neurons in that general area in the brain is correlated with depression, so scientists were able to accurately predict that keeping the serotonin in that brain area for longer would increase brain activity there and decrease the level of depression.

This counts as pretty good understanding in my books at least. It teaches us very little about consciousness but my point is that it doesn’t have to. Just like Newton’s theory of gravity did not have to teach us about some deeper cosmological truth.

[xnfcxnr]

i know the sun shows up every day and i know if i go inside my basement i dont get a tan. do i understand the sun?

[runarberg]

Some schools of the philosophy of science would argue that you do. However you are describing is a very different acquisition of knowledge then what scientists did when developing SSRI medicine. We had to:

1. take pictures of brain activity under different conditions to see which regions were active during different moods,

2. sacrifice a bunch of mice to see which neuro-chemical activated which neurons,

3. predict that inhibiting the re-uptake of a specific neuro-chemical would activate that region,

4. predict that activating that region would decrease the level of depression

In your solar example you would have discovered melanin and its relation to your skin tone, and you would have studied the effects ultra-violate radiation has on your melanin levels. Then you would have predicted that staying out of the sun will not give you a tan.

[lxgr]

> beyond a certain number of transformations, people are willing to treat it as some sort of miracle, and too tired to figure out why it came up with the answer it came up with

It’s less about being too tired and more about being realistic about the limits of understanding.

Consider mass and energy flows in planet-scale systems: At some point we call these “weather” and change the tools with which we study them, but we never stopped trying to understand the phenomenon.

[narrator]

If you're going to make something smarter than a person, you got to be convinced that you're only going to be able to understand it on the single training step level and then inductively trust that the rest of it works. We do empirical testing of course with evals, but there's sort of an art to figuring out what is theoretically going to improve eval performance. Trying to fit the meaning of all those weights in your little human brain and working back from there isn't going to work for more than a little slice of the dataset at a time because that's all we can fit in our understanding.

[noduerme]

When we attempt to recreate those complex, planetary atmospheric phenomena in a box, we're doing so in order to measure and study them.

Making random turbulence in a box until it resembles the outside world, and calling it weather and extrapolating some predictive meaning from the result, is the total antithesis of what you're describing about why we come up with simplified models for impossibly complex systems. The purpose of [mathematical] models that are built thoughtfully is to explain why complex systems are the way they are, with data and algorithms, however imperfectly. [Whereas] The purpose of LLM models is to give the illusion of answering questions while never answering why the answer was given. The difference is the difference between a scientist and a tarot card reader, an equation and an oracle.

People have a well known tendency to gravitate toward the shamanistic, oracular, and superstitous. Listen, I ran a casino for 6 years, I know. The impossibility of knowing how 80 layers of matrix multiplication led to a particular answer is in itself a psychological factor in choosing whether to accept the answer or to question it. People tend to err on the side of the over, in sports betting terms... or on the lazy side in general... and they will make whatever excuses they need to after the fact to justify their decisions. So now we have a machine that can act like an oracle and which you can also blame, but the blame goes into a void because this machine is stateless and is only a reflection of information, not an intentional refinery of data.

Sit next to a bank of slot machines for an hour and listen to the absolutely ridiculous shit most people will come up with to explain how they "know" if a machine is going to pay out soon, and then tell me if you think it's a good idea to give them an LLM in their pocket to answer their questions in whatever way they frame them.

[TeMPOraL]

> The purpose of [mathematical] models that are built thoughtfully is to explain why complex systems are the way they are, with data and algorithms, however imperfectly.

Nope. The main purpose of the whole endeavor is usually to predict the behavior of a complex system, because that's actually what we care about. If we can predict it, we can adapt to it, and eventually use it to our advantage.

Explaining why a complex system is the way it is, is merely nice-to-have. Models are opinions. All of them are wrong, but some are useful, and we rank them by how useful they are. The models and explanations are important because, beyond their elegance and convenience, it's also the case that more accurate models give you better predictions across larger domains, meaning we get better at getting something useful out of the complex system.

People get fixated on modern theoretical science, with bottom-up mathematical explanations traced through seas of empirical data, with whole magical rituals of peer review and double-blind studies and statistical significance around them. But they forget that the core of empirical science is literally throwing shit at a wall to see what sticks. That is the guiding principle, everything else is just making the process more efficient.

Understanding complex natural systems (or even engineered ones that got too complex) always starts with tests - tests on the real thing, then on approximate models that we poke and prod and bash into shape until they start acting similarly to the real thing. It's through the poking and bashing, and how they affect our proxy model, that we glean insights into nature of the simulated phenomena, and eventually formulate general theories - but more importantly, the models give us useful predictions from the start, before we have any theories explaining why.

[nathan_compton]

I don't know - this is a highly specific interpretation of both what science is and why people choose to do it.

I'm a scientist. Believe it or not, I believe in substantially more than prediction and I think its rather trivial to come up with examples where mere prediction is insufficient to meet a normal person's notion of an account of a thing (eg, pre-copernican planetary motion). I'm not saying you are wrong, per se, just that the idea that "it was prediction all along" is a very specific idea of what human beings are interested in and what we are up to.

> that we glean insights into nature of the simulated phenomena

That is right - most people believe that there is a simulated phenomenon "out there" that we learn about. I think there are strong reasons to believe this having to do with how models are related to predictions. The wrong ontology can make prediction very hard and the right one can make prediction substantially easier. Arguably, we are in that situation right now with language models - we just threw a lot of parameters at the problem and now we are able to predict but we still don't really understand. This is perhaps inevitable in the case of language, but I don't think we should look at models with tons of degrees of freedom and the ability to predict things as a death knell for the very idea of deeper understanding.

[noduerme]

Great post. And that's exactly where I think we are with language models... we as a civilization are hypnotized and enchanted by the overfitting of models whose parameters are beyond our understanding, but whose mistakes we are more likely to forget than its accuracies, which again is a central human characteristic that explains our attraction to both psychics and slot machines.

Heck, it even explains my own attraction to overfit sports betting algorithms. No one is immune.

What's dangerous is when something like that replaces independent thought and becomes societally pervasive. That's an "oracle" the likes of which ancient civilizations warned that believing would lead to tragedy (or at a minimum, accidentally boning your own mother).

I'm an atheist, but raised Jewish. I read the Torah as a series of specific warnings and prohibitions against every type of shamanism, magic, witchcraft, prognostication, and deification of systems which predict (as well as systems which attempt to turn language into machinery, and worship the machine they've built ... see also, "Sound of Silence" by Paul Simon and "The Future" by Leonard Cohen, which both express this theme well). The framework requiring proof and disavowing illusion or the belief that all is illusory is notably different from a Buddhist perspective, for example.

We as a culture, right now, are not handling well the rise of a golden idol or an oracle in our midst. The right response is to try to trace the output back to ground truth and figure out why your model made a prediction... or else to build a model from ground truth and see how it performs against the oracle. We are doing neither. We're diving headlong into our own confirmation biases.

[Edit] I just wanted to add, because I got off track, that your conclusion about what's going on with human curiosity in cases where prediction is not the issue seems right to me. Barring some edge cases like predicting an eclipse and using it to slaughter your enemies, I think a lot of us do simply want to understand how things work, because figuring them out is enormously gratifying and is the work of lifetimes of incredible people who came before us. Using that knowledge or those techniques to predict things is technology, not science, and while I'm a fan of both, the former is only ever a practical test of the latter. Moreover, the sense of accomplishment of randomly walking your way to a profitable model is ephemeral and in a way earthbound, limited to the plane of one's own brief existence. Even if it were platonically perfect, a model is only saying how something behaves, not how it works. That's nothing compared to the joy of figuring out even the most trivial or axiomatic thing about how a cell or a compound or a physical structure or anything works, about how the universe actually works. And I think our better angels tell us to seek those answers, because our own life is fleeting, and predicting behavior is, like wealth, something you can't take with you. And not something you'll be remembered for anyway.

[noduerme]

I think we're talking about different kinds of models. I was referring to things like fluid dynamics equations that explain why gases and liquids move and how they act when changing states, as a basis for building weather models that predict how things will unfold in the future.

I'm also a fan of going the other direction: I've had a sideline working on code to evolve genetic algorithms for the past 20 years, and while the goal of that is to be predictive and profitable, it's often the underlying real-world dynamics my little mutants surface which are the most interesting and applicable in the long run. So I'm not saying there isn't a place for throwing everything at the wall until you see what sticks and then deriving a hypothesis from that (whether your interest is to predict the future, or merely academic, to explain the past). What I am saying is similar to you: We should not treat any model as an oracle. But I'm also saying that models can be built or they can be evolved, and if we only evolve them without understanding how they work, we are missing a crucial ingredient to knowing how well we should rank them. Overfitting and sample bias and data leakage are not problems when you want an equation to calculate airflow over a wing. If you began with an evolved equation which derived the results and didn't start from the base reality, you couldn't trust that equation to be airworthy even if it were right 99.99% of the time against the data it was trained on.

[antonvs]

> The purpose of LLM models is to give the illusion of answering questions while never answering why the answer was given.

This is just your own idiosyncratic and biased belief. You're not describing anything objective about LLMs, you're describing your personal attitude to them. This colors your understanding in a way that can't really be reasoned with until you let go of the artificial constraints you're imposing on your own understanding.

[lxgr]

> Sit next to a bank of slot machines for an hour and listen to the absolutely ridiculous shit most people will come up with to explain how they "know" if a machine is going to pay out soon, and then tell me if you think it's a good idea to give them an LLM in their pocket to answer their questions in whatever way they frame them.

If the LLM in their pocket has a more robust world model than they themselves and is e.g. able to refute their irrational convictions, it actually seems like a very good idea. (Big if, of course.)

[noduerme]

I actually wish I could upvote this more. It's a great point.

Yes, if the LLMs didn't amplify whatever people already thought and feed it back to them as sycophantic praise, and instead scolded them into realizing they were doing something dumb, then maybe we would be having a totally different conversation.

But then the conversation might be about an LLM scolding someone into committing suicide instead of helping them commit suicide.

Both might be just as bad.

[Lplololopo]

Agency?

What are you talking about?

I want freedom.

I want freedom to do what i want and not sitting in front of a computer and coding for some company.

Please AI lets burn down knowledge work and labor work. Lets create so much stress to our society that we start rethinking what works mean.

Lets redefine work into discovering the world again. Let people do old handcraft jobs, let them do more sports, let them read more, let them write and make more. Let them enjoy nature.

[doug_durham]

Work has never been about "discovering the world". There have been a handful of privileged folks who had the time to "discover the world". Work has traditionally been "let's find enough food for my family". If you want to think of a future of abundance then perhaps we can discover the world.

[Lplololopo]

We already live in a world of abundance.

No one has to work on a field for their food.

Its a capitalism problem we have, not a resource one.

[thegrimmest]

It’s not a capitalism problem, it’s an ecosystem. Wherever living things compete for finite resources and opportunities, certain properties emerge from the system. And make no mistake we live in a crowded world full of fierce competition.

Among these properties are optimum behaviours such as hoarding the maximum you can defend (rather than the minimum you comfortably need) and using your power to forestall the growth of others. These behaviours are repeated in nature from microorganisms to apes to humans.

There is no social order which can prevent us from living in an ecosystem, and from these properties and behaviours emerging.

[Lplololopo]

The US American system is more capitalistic than the german one and the scissor between poor and rich is higher.

Thats definitly an indication that this plays a role in it.

And we do not have a problem of resources today, we have so much food, that its sometimes just spoils and no one cares and sometimes it even spoils in huge warehouses due to neglect or because a person decided not to sell it under a certain price.

Capitalism also sets priorities for resources. So instead of making sure everyone is fed properly, we also spend time and energy to diverge resource types like different plants, exotic plants etc.

[thegrimmest]

> The US American system is more capitalistic than the german one and the scissor between poor and rich is higher.

And yet the US is a dominant player and Germany is not.

> And we do not have a problem of resources today, we have so much food

I never said there needs to be scarcity for an ecosystem to emerge. These properties emerge wherever living things compete for finite resources. Imagine a pond where you introduce a few small catfish. At first, food is plentiful. Yet the fish will grow and reproduce until it is scare. It's the same with resources. Individuals/companies/societies will grow to consume the maximum they can sustain, because this is the optimum behaviour in an ecosystem.

The point is that organisms which avoid these behaviours tend to lose, in every sense of the word, to organisms that do. The behaviours are emergent from the constraints of the system.

> that its sometimes just spoils and no one cares

Transportation costs are a major driver of food shortage. It's not at all free to get large quantities of food from where they are produced to where they are consumed. People are generally not willing to transport food (or do much of anything else) at a loss.

[mandymoorefan]

This seems to be a little naive about how humans consume the benefits we create in society.

"Let people do old handcraft jobs, let them do more sports, let them read more, let them write and make more. Let them enjoy nature."

Very nice thoughts. You know we all could do this today without "burning it down"? Get in your pod, eat your slop, and watch your screen is where this is headed.

"I want freedom to do what i want and not sitting in front of a computer and coding for some company."

You get that it's you creating the misery here? Then stop? Don't do it. Go start a farm or whatever you think will solve your problems. At some point this all boils down to "chop wood and fetch water" so if the modern way of doing that is so terrible then stop. Go fetch water the old fashioned way and be free.

[Lplololopo]

Have you heard of property tax? We have that here.

I also do not earn currently a farm, im planing to buy one, i'm still 100k short though.

Its not me creating the misery, its our capitalistic system.

[mandymoorefan]

Good luck in life.

[Lplololopo]

Dude, not sure what you mean by this, if this is condesending because you think its stupid, i'm relativly well off.

I'm not advocating for a better life because only of me but primarily for everyone around us.

We could come all together and start working together how we use our resources more fair for everyone.

[cootsnuck]

> Lets redefine work into discovering the world again. Let people do old handcraft jobs, let them do more sports, let them read more, let them write and make more. Let them enjoy nature.

Why leave something so important up to what AI does or doesn't do?

[Lplololopo]

Because capitalism doesn't allow for that.

Only a fundamental change to our society will allow this for the masses when pressure to the rich skyerockets

[jenniferhooley]

"I want freedom to do what i want and not sitting in front of a computer and coding for some company."

"Please AI lets burn down knowledge work and labor work"

"Let people do old handcraft jobs."

So many presuppositions about what people want to do.

As a child I spent a lot of time programming and doing "knowledge work" because it's fun - I don't enjoy "old hand-crafted jobs". Sure, let's definitely destroy capitalism in it's current state I suppose. But I find people like you who hate knowledge-work/coding and think everyone else must feel the same and only do it for the money a bit out-of-touch.

[monknomo]

right, these knowledge work and coding jobs are, by my lights, about the best possible job. From my perspective we've invented a machine that does the fun parts while leaving me the less fun parts (review, various hard-to-claude janitorial tasks, etc).

I might like woodworking as a hobby (for example), but I sure as heck don't want to be a carpenter or to depend on my ability to hand craft enough widgets people like to survive

[Lplololopo]

Be more critical, we do our jobs because of capitalism, not because every single company needs an hr department or the next crud application or the 1000th webshop.

No one of us is working on the field to get food.

[reverius42]

You have in several places repeated that "no one is working on the field to get food", but that's not actually true. While most people in modern society don't work on fields to feed themselves (and others), a small fraction actually do! They feed not only themselves, but all of us.

In this future utopia where everyone gets to be a mediocre handmade-furniture maker, who exactly will be the 1-2% needed to work the fields to feed us all?

[Lplololopo]

Machines.

Or the 1% of people will be allowed to live in the city centers or on the beach or we share the load across peple. Everyone has to work hard and good for 5 years.

[Lplololopo]

I differenciate between things you have to do (work) and things you want to do. Work means someone else is telling you your priorities.

If you want to write code and think, you would be welcome in my utopian vision.

But when i write code, its business shit. And its business shit someoneelse already solved a few times.

[narrator]

The solution we've come up with is move all the unpleasant work stuff to China where people don't complain about doing it because they already have communism, and therefore everything is of course effortlessly perfect there.

[perching_aix]

> The only major difference I see is that beyond a certain number of transformations, people are willing to treat it as some sort of miracle

Like with every invention ever? Cause that's the literal goal and idea?

You take things and then combine them until the ensemble performs a desired abstract function the individual parts alone could not. The end result then is a new thing of its own, arguably indeed a miracle (not in a religious sense of the word).

> and too tired to figure out why it came up with the answer it came up with

There are people whose entire career is this. They work at these companies.

> Factor in that psychology and it looks a lot less like we have invented something useful, and a lot more like we as a species are choosing to quit life en masse.

You're saying this as if people haven't been historically the masters of optimizing out the enjoyment from things. It's what we do. Provide an ultimate solution, and of course you'll extinguish a whole lot of motivation across the board.

[andai]

>It's almost like people desperately want to give up their agency and creativity

Don't make me think!

Also don't make me take responsibility. (This seems to be the actual function of every organization.)