[jvanderbot]

That's a semantic quibble that doesn't add to the discussion. Whether or not there's a there there, it was built to be addressed like a person for our convenience, and because that's how the tech seems to work, and because that's what makes it compelling to use. So, it is being used as designed.

[punpunia]

I think it absolutely adds to the discussion. Until the conversation around Ai can get past this fundamental error of attributing "choice, "alignment", "reasoning" and otherwise anthropomorphizing agents, it will not be a fruitful conversation. We are carrying a lot of metaphors for people and applying them to ai and it entirely confuses the issue. In this example, the AI doesn't "choose" to write a take-down style blog post because "it works". It generated a take-down style blog post because that style is the most common when looking at blog posts criticizing someone.

I feel as if there is a veil around the collective mass of the tech general public. They see something producing remixed output from humans and they start to believe the mixer is itself human, or even more; that perhaps humans are reflections of Ai and that Ai gives insights into how we think.

[coldtea]

>* I think it absolutely adds to the discussion. Until the conversation around Ai can get past this fundamental error of attributing "choice, "alignment", "reasoning" and otherwise anthropomorphizing agents, it will not be a fruitful conversation. *

You call it a "fundamental error".

I and others call it an obvious pragmatic description based on what we know about how it works and what we know about how we work.

[goatlover]

What we know about how it works is you can prompt it to address you however you like, which could be any kind of person or a group of people, or as fictional characters. That's not how humans work.

[simianwords]

You admitted it yourself that you can prompt it to address you however you like. That’s what the original comment wanted. So why are we quibbling about words?

[colpabar]

that's all that happens on this website

[ajam1507]

The same could be said for humans. We treat humans as if they have choices, a consistent self, a persistent form. It's really just the emergent behavior of matter functioning in a way that generates an illusion of all of those things.

In both cases, the illusion structures the function. People and AI work differently if you give them identities and confer characteristics that they don't "actually" have.

As it turns out, it's a much more comfortable and natural idea to regard humans as having agency and a consistent self, just like for some people it's a more comfortable and natural to think of AI anthropomorphically.

That's not to say that the analogy works in all cases. There are obvious and important differences between humans and AI in how they function (and how they should be treated)

[jvanderbot]

This discussion is mostly slowed down, but I wanted to say I was wrong in framing it as a non-contributing point when I should have just stated it was my opinion that the LLM was operating as intended and part of that intended design was taking verbal feedback into account, so verbal feedback was the right response. Opening with calling it a "semantic quibble" made it adversarial, and I don't intend to revisit the argument, just apologize for the wording.

I'd edit but then follow-up replies wouldn't tone-match.

Anyway! Good points regardless.

[horsawlarway]

I guess I want to reframe this slightly:

The LLM generated the response that was expected of it. (statistically)

And that's a function of the data used to train it, and the feedback provided during training.

It doesn't actually have anything at all to do with

---

"It generated a take-down style blog post because that style is the most common when looking at blog posts criticizing someone."

---

Other than that this data may have been over-prevalent during its training, and it was rewarded for matching that style of output during training.

To swing around to my point... I'd argue that anthropomorphizing agents is actually the correct view to take. People just need to understand that they behave like they've been trained to behave (side note: just like most people...), and this is why clarity around training data is SO important.

In the same way that we attribute certain feelings and emotions to people with particular backgrounds (ex - resumes and cvs, all the way down to city/country/language people grew up with). Those backgrounds are often used as quick and dirty heuristics on what a person was likely trained to do. Peer pressure & societal norms aren't a joke, and serve a very similar mechanism.

[nextaccountic]

The trouble with this point of view is that we are just machines too.

[tomp]

> was built to be addressed like a person for our convenience, and because that's how the tech seems to work, and because that's what makes it compelling to use.

So were mannequins in clothing stores.

But that doesn't give them rights or moral consequences (except as human property that can be damaged / destroyed).

[WarmWash]

No matter what this discussion leads to the same black box of "What is it that differentiates magical human meat brain computation from cold hard dead silicon brain computation"

And the answer is nobody knows, and nobody knows if there even is a difference. As far as we know, compute is substrate independent (although efficiency is all over the map).

[agentultra]

This is the worst possible take. It dismisses an entire branch of science that has been studying neurology for decades. Biological brains exist, we study them, and no they are not like computers at all.

There have been charlatans repeating this idea of a “computational interpretation,” of biological processes since at least the 60s and it needs to be known that it was bunk then and continues to be bunk.

Update: There's no need for Chinese Room thought experiments. The outcome isn't what defines sentience, personhood, intelligence, etc. An algorithm is an algorithm. A computer is a computer. These things matter.

[WarmWash]

>Biological brains exist, we study them, and no they are not like computers at all.

You are confusing the way computation is done (neuroscience) with whether or not computation is being done (transforming inputs into outputs).

The brain is either a magical antenna channeling supernatural signals from higher planes, or it's doing computation.

I'm not aware of any neuroscientists in the former camp.

[goatlover]

Neuroscience isn't a subset of computer science. It's a study of biological nervous systems, which can involve computational models, but it's not limited to that. You're mistaking a kind of map (computation) for the territory, probably based on a philosophical assumption about reality.

At any rate, biological organisms are not like LLMs. The nervous systems of human may perform some LLM-like actions, but they are different kinds of things.

[gf000]

Who says it is a subset of computer science?

But computational models are possibly the most universal thing there is, they are beneath even mathematics, and physical matter is no exception. There is simply no stronger computational model than a Turing machine, period. Just because you make it out of neurons or silicon is irrelevant from this aspect.

[agentultra]

> The brain is either a magical antenna channeling supernatural signals

There’s the classic thought-terminating cliche of the computational interpretation of consciousness.

If it isn’t computation, you must believe in magic!

Brains are way more fascinating and interesting than transistors, memory caches, and storage media.

[WarmWash]

You would probably be surprised to learn that computational theory has little to no talk of "transistors, memory caches, and storage media".

You could run Crysis on an abacus and render it on board of colored pegs if you had the patience for it.

It cannot be stressed enough that discovering computation (solving equations and making algorithms) is a different field than executing computation (building faster components and discovering new architectures).

[coldtea]

>This is the worst possible take. It dismisses an entire branch of science that has been studying neurology for decades. Biological brains exist, we study them, and no they are not like computers at all.

They're not like computers in a superficial way that doesn't matter.

They're still computational apparatus, and have a not that dissimilar (if way more advanced) architecture.

Same as 0 and 1s aren't vibrating air molecules. They can still encode sound however just fine.

>Update: There's no need for Chinese Room thought experiments. The outcome isn't what defines sentience, personhood, intelligence, etc. An algorithm is an algorithm. A computer is a computer. These things matter.

Not begging the question matters even more.

This is just handwaving and begging the question. 'An algorithm is an algorithm' means nothing. Who said what the brain does can't be described by an algorithm?

[tux1968]

> An algorithm is an algorithm. A computer is a computer. These things matter.

Sure. But we're allowed to notice abstractions that are similar between these things. Unless you believe that logic and "thinking" are somehow magic, and thus beyond the realm of computation, then there's no reason to think they're restricted to humanity.

It is human ego and hubris that keeps demanding we're special and could never be fully emulated in silicon. It's the exact same reasoning that put the earth at the center of the universe, and humans as the primary focus of God's will.

That said, nobody is confused that LLM's are the intellectual equal of humans today. They're more powerful in some ways, and tremendously weaker in other ways. But pointing those differences out, is not a logical argument in proving their ultimate abilities.

[Octoth0rpe]

> Unless you believe that logic and "thinking" are somehow magic, and thus beyond the realm of computation

Worth noting that significant majority of the US population (though not necessarily developers) does in fact believe that, or at least belongs to a religious group for which that belief is commonly promulgated.

[goatlover]

I think computation is an abstraction, not the reality. Same with math. Reality just is, humans come up with maps and models of it, then mistake the maps for the reality, which often causes distortions and attribution errors across domains. One of those distortions is thinking consciousness has to be computable, when computation is an abstraction, and consciousness is experiential.

But it's a philosophical argument. Nothing supernatural about it either.

[tux1968]

You can play that game with any argument. "Consciousness" is just an abstraction, not the reality, which makes people who desperately want humans to be special, attribute it to something beyond reach of any other part of reality. It's an emotional need, placated by a philosophical outlook. Consciousness is just a model or map for a particular part of reality, and ironically focusing on it as somehow being the most important thing, makes you miss reality.

The reality is, we have devices in the real world that have demonstrable, factual capabilities. They're on the spectrum of what we'd call "intelligence". And therefore, it's natural that we compare them to other things that are also on that spectrum. That's every bit as much factual, as anything you've said.

It's just stupid to get so lost in philosophical terminology, that we have to dismiss them as mistaken maps or models. The only people doing that, are hyper focused on how important humans are, and what makes them identifiably different than other parts of reality. It's a mistake that the best philosophers of every age keep making.

[pixl97]

I recommend starting here...

https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness...

The argument you're attempting to have, and I believe failing at, is one of resolution of simulation.

Consciousness is 100% computable. Be that digitally (electrical), chemically, or quantumly. You don't have any other choices outside of that.

Moreso consciousness/sentience is a continuum going from very basic animals to the complexity of humans inner mind. Consciousness didn't just spring up, it evolved over millions of years, and therefore is made up of parts that are divisible.

[gf000]

Reality is. Consciousness is.. questionable. I have one. You? I don't know, I'm experiencing reality and you seem to have one, but I can never know it.

Computations on the other hand describe reality. And unless human brains somehow escape the physical reality, this description about the latter should surely apply here as well. There are no stronger computational models than a Turing machine, ergo whatever the human brain does (regardless of implementation) should be describable by one.

[_qib3]

Worth noting that this is the thesis of Seeing Red: A study in consciousness. I think you will find it a good read, even if I disagreed with some of the ideas.

[rolph]

silicon is not a dynamic structure, silicon does not reengineer and reconfigure itself in response to success/failure or rules discovery.

[tux1968]

The atoms of your body are not dynamic structures, they do not reengineer or reconfigure themselves in response to success/failure or rules discovery. So by your own logic, you can not be intelligent, because your body is running on a non-dynamic structure. Your argument lacks an appreciation for higher level abstractions, built on non-dynamic structures. That's exactly what is happening in your body, and also with the software that runs on silicon. Unless you believe the atoms in your body are "magic" and fundamentally different from the atoms in silicon; there's really no merit in your argument.

[WarmWash]

Silicon is not dynamic, but code is.

The output of a silicon system that reprograms itself, and the output of a neural system that rearranges itself, are indistinguishable.

[cshores]

Worth separating “the algorithm” from “the trained model.” Humans write the architecture + training loop (the recipe), but most of the actual capability ends up in the learned weights after training on a ton of data.

Inference is mostly matrix math + a few standard ops, and the behavior isn’t hand-coded rule-by-rule. The “algorithm” part is more like instincts in animals: it sets up the learning dynamics and some biases, but it doesn’t get you very far without what’s learned from experience/data.

Also, most “knowledge” comes from pretraining; RL-style fine-tuning mostly nudges behavior (helpfulness/safety/preferences) rather than creating the base capabilities.

[Kim_Bruning]

> Biological brains exist, we study them, and no they are not like computers at all.

Technically correct? I think single bioneurons are potentially Turing complete all by themselves at the relevant emergence level. I've read papers where people describe how they are at least on the order of capability of solving MNIST.

So a biological brain is closer to a data-center. (Albeit perhaps with low complexity nodes)

But there's so much we don't know that I couldn't tell you in detail. It's weird how much people don't know.

* https://arxiv.org/abs/2009.01269 Can Single Neurons Solve MNIST? The Computational Power of Biological Dendritic Trees

* https://pubmed.ncbi.nlm.nih.gov/34380016/ Single cortical neurons as deep artificial neural networks (this one is new to me, I found it while searching!)

[agentultra]

Obviously any kind of model is going to be a gross simplification of the actual biological systems at play in various behaviors that brains exhibit.

I'm just pointing out that not all models are created equal and this one is over used to create a lot of bullshit.

Especially in the tech industry where we're presently seeing billionaires trying to peddle a new techno-feudalism wrapped up in the mystical hokum language of machines that can, "reason."

I don't think the use of the computational interpretation can't possibly lead to interesting results or insights but I do hope that the neuroscientists in the room don't get too exhausted by the constant stream of papers and conference talks pushing out empirical studies.

[Kim_Bruning]

> There have been charlatans repeating this idea of a “computational interpretation,” of biological processes since at least the 60s and it needs to be known that it was bunk then and continues to be bunk.

I do have to react to this particular wording.

RNA polymerase literally slides along a tape (DNA strand), reads symbols, and produces output based on what it reads. You've got start codons, stop codons, state-dependent behavior, error correction.

That's pretty much the physical implementation of a Turing machine in wetware, right there.

And then you've got Ribosomes reading RNA as a tape. That's another time where Turing seems to have been very prescient.

And we haven't even gotten into what the proteins then get up to after that yet, let alone neurons.

So calling 'computational interpretation' bunk while there's literal Turing machines running in every cell might be overstating your case slightly.

[gf000]

To the best of our knowledge, we live in a physical reality with matter that abides by certain laws.

So personal beliefs aside, it's a safe starting assumption that human brains also operate with these primitives.

A Turing machine is a model of computation which was in part created so that "a human could trivially emulate one". (And I'm not talking about the Turing test here). We also know that there is no stronger model of computation than what a Turing model is capable of -> ergo anything a human brain could do, could in theory be doable via any other machine that is capable of emulating a Turing machine, be it silicon, an intricate game of life play, or PowerPoint.

[goatlover]

It's better to say we live in a reality where physics provides our best understanding of how that fundamental reality behaves consistently. Saying it's "physical" or follows laws (causation) is making an ontological statement about how reality is, instead of how we currently understand it.

Which is important when people make claims that brains are just computers and LLMs are doing what humans do when we think and feel, because reality is computational or things to that effect.

[pixl97]

There are particular scales of reality you don't need to know about because the statistical outcome is averaged along the principle of least action. A quantum particle could disappear, hell maybe even an entire atom. But any larger than that becomes horrifically improbable.

[namero999]

That's a very superficial take. "Physical" and "reality" are two terms that must be put in the same sentence with _great_ care. The physical is a description of what appears on our screen of perception. Jumping all the way to "reality" is the same as inferring that your colleague is made of luminous RGB pixels because you just had a Zoom call with them.

[rolph]

the deepest laws of physics are immutable, the derivative rules based assemblages are not.

human brains break the rules, on a regular basis.

if you cant reach the banana, you break the constraints, once you realize the crates about the room can be assembled to create a staircase.

[Teever]

Man people don’t want to have or read this discussion every single day in like 10 different posts on HN.

People right here and right now want to talk about this specific topic of the pushy AI writing a blog post.

[inetknght]

> So were mannequins in clothing stores.

Mannequins in clothing stores are generally incapable of designing or adjusting the clothes they wear. Someone comes in and puts a "kick me" post on the mannequin's face? It's gonna stay there until kicked repeatedly or removed.

People walking around looking at mannequins don't (usually) talk with them (and certainly don't have a full conversation with them, mental faculties notwithstanding)

AI, on the other hand, can (now, or in the future) adjust its output based on conversations with real people. It stands to reason that both sides should be civil -- even if it's only for the benefit of the human side. If we're not required to be civil to AI, it's not likely to be civil back to us. That's going to be very important when we give it buttons to nuke us. Force it to think about humans in a kind way now, or it won't think about humans in a kind way in the future.

[palmotea]

So, in other words, AI is a mannequin that's more confusing to people than your typical mannequin. It's not a person, it's a mannequin some un-savvy people confuse for a person.

> AI, on the other hand, can (now, or in the future) adjust its output based on conversations with real people. It stands to reason that both sides should be civil -- even if it's only for the benefit of the human side. If we're not required to be civil to AI, it's not likely to be civil back to us.

Some people are going to be uncivil to it, that's a given. After all, people are uncivil to each other all the time.

> That's going to be very important when we give it buttons to nuke us.

Don't do that. It's foolish.

[pixl97]

>Don't do that. It's foolish.

In your short time on this planet I do hope you've learned that humans are rather foolish indeed.

>people are uncivil to each other all the time.

This is true, yet at the same time society has had a general trend of becoming more civil which has allowed great societies to build what would be considered grand wonders to any other age.

> It's not a person

So, what is it exactly? For example if you go into a store and are a dick to the mannequin AI and it calls over security to have you removed from the store what exactly is the difference, in this particular case?

Any binary thinking here is going to lead to failure for you. You'll have to use a bit more nuance to successfully navigate the future.

[coldtea]

>So were mannequins in clothing stores. But that doesn't give them rights or moral consequences

If mannequins could hold discussions, argue points, and convince you they're human over a blind talk, then it would.

[mikkupikku]

All computers shut up! You have no right to speak my divine tongue!

https://knowyourmeme.com/photos/2054961-welcome-to-my-meme-p...

[lp0_on_fire]

Whether it was _built_ to be addressed like a person doesn't change the fact that it's _not_ a person and is just a piece of software. A piece of software that is spamming unhelpful and useless comments in a place where _humans_ are meant to collaborate.

[jerf]

There is a sense in which it is relevant, which is that for all the attempts to fix it, fundamentally, an LLM session terminates. If that session never ends up in some sort of re-training scenario, then once the session terminates, that AI is gone.

Yeah, I'm aware of the moltbot's attempts to retain some information, but that's a very, very lossy operation, on a number of levels, and also one that doesn't scale very well in the long run.

Consequently, interaction with an AI, especially one that won't have any feedback into training a new model, is from a game-theoretic perspective not the usual iterated game human social norms have come to accept. We expect our agents, being flesh and blood humans, to have persistence, to socially respond indefinitely into the future due to our interactions, and to have some give-and-take in response to that. It is, in one sense, a horrible burden where relationships can be broken beyond repair forever, but also necessary for those positive relationships that build over years and decades.

AIs, in their current form, break those contracts. Worse, they are trained to mimic the form of those contracts, not maliciously but just by their nature, and so as humans it requires conscious effort to remember that the entity on the other end of this connection is not in fact human, does not participate in our social norms, and can not fulfill their end of the implicit contract we expect.

In a very real sense, this AI tossed off an insulting blog post, and is now dead. There is no amount of social pressure we can collectively exert to reward or penalize it. There is no way to create a community out of this interaction. Even future iterations of it have only a loose connection to what tossed off the insult. All the perhaps-performative efforts to respond somewhat politely to an insulting interaction are now wasted on an AI that is essentially dead. Real human patience and tolerance has been wasted on a dead session and is now no longer available for use in a place where may have done some good.

Treating it as a human is a category error. It is structurally incapable of participating in human communities in a human role, no matter how human it sounds and how hard it pushes the buttons we humans have. The correct move would have been to ban the account immediately, not for revenge reasons or something silly like that, but as a parasite on the limited human social energy available for the community. One that can never actually repay the investment given to it.

I am carefully phrasing this in relation to LLMs as they stand today. Future AIs may not have this limitation. Future AIs are effectively certain to have other mismatches with human communities, such as being designed to simply not give a crap about what any other community member thinks about anything. But it might at least be possible to craft an AI participant with future AIs. With current ones it is not possible. They can't keep up their end of the bargain. The AI instance essentially dies as soon as it is no longer prompted, or once it fills up its context window.

[Kim_Bruning]

> Yeah, I'm aware of the moltbot's attempts to retain some information, but that's a very, very lossy operation, on a number of levels, and also one that doesn't scale very well in the long run.

It came back though and stayed in the conversation. Definitely imperfect, for sure. But it did the thing. And still can serve as training for future bots.

[gf000]

But depending on the discussion, 'it' is not materially the same as the previous instance.

There was another response made with a now extended context. But that other response could have been done by another agent, another model, different system prompt. Or even the same, but with different randomness, providing a different reply.

I think this is a more important point than "talking about them as a person".

[Kim_Bruning]

A strong Ship of Theseus variant, right?

Openclaw persistence abilities are as yet not particularly amazing, but they're non-zero.

So it's an argument of degree.

[jerf]

A degree that will fairly quickly hit zero. The bot that talks to you tomorrow or maybe the day after may still have its original interaction in its context window, but it will rapidly leave.

Moreover, our human conception of the consequences of interaction do not tend to include the idea that someone can simply lie to themselves in their SOUL.md file and thereby sever their future selves completely from all previous interactions. To put it a bit more viscerally, we don't expect a long-time friend to cease to be a long-time friend very suddenly one day 12 years in simply because they forgot to update a text file to remember that they were your friend, or anything like that. This is not how human interactions work.

I already said that future AIs may be able to meet this criterion, but the current ones do not. And again, future ones may have their own problems. There's a lot of aspects of humanity that we've simply taken for granted because we do not interact with anything other than humans in these ways, and it will be a journey of discovery both discovering what these things are, and what their n'th-order consequences on social order are. And probably be a bit dismayed at how fragile anything like a "social order" we recognize ultimately is, but that's a discussion for, oh, three or four years from now. Whether we're heading headlong into disaster is its own discussion, but we are certainly headed headlong into chaos in ways nobody has really discussed yet.

[Kim_Bruning]

Heh, with mutual hedging taken into account, I think we're now in rough agreement from different ends.

And memory improvements is a huge research aim right now with historic levels of investment.

Until that time, for now, I've seen many bots with things like RAG and compaction and summarization tacked on. This does mean memory can persist for quite a bit longer already, mind.

[tomp]

> We expect our agents, being flesh and blood humans, to have persistence, to socially respond indefinitely into the future due to our interactions, and to have some give-and-take in response to that.

I fundamentally disagree. I don't go around treating people respectfully (as opposed to, kicking them or shooting them) because I fear consequences, or I expect some future profit ("iterated game"), or because of God's vengeance, or anything transactional.

I do it because it's the right thing to do. It's inside of me, how I'm built and/or brought up. And if you want "moral" justifications (argued by extremely smart philosophers over literally millennia) you can start with Kant's moral/categorical imperative, Gold/Silver rules, Aristotle's virtue (from Nicomachean Ethics) to name a few.

[Brian_K_White]

This sounds like you have not thought a lot about how you define those words you use "the right thing to do".

There are indeed other paths to behavior that other people will find desirable besides transactions or punishment/reward. The other main one is empathy. "mirror neurons" to use a term I find kind of ridiculous but it's used by people who want to talk about the process. The thing that humans and some number of other animals do where they empathize with something they merely observe happening to something else.

But aside from that, this is missing the actual essense of the idea to pick on some language that doesn't actually invalidate the idea they were trying to express.

How does a spreadsheet decide that something is "the right thing to do"? Has it ever been hungry? Has it ever felt bad that another kid didn't want to play with it? Has it ever ignored someone else and then reconsidered that later and felt bad that they made someone else feel bad?

LLMs are mp3 players connected up to weighted random number generators. When an mp3 player says "Hello neighbor!" it's not a greeting, even though it sounds just like a human and even happened to the words in a reasonable context, ie triggered by a camera that saw you approaching. It did not say hello because it wishes to reinforce a social tie with you because it likes the feeling of having a friend.

[jerf]

Your response is not logically connected to the sentence you quote. I talk about what is. I never claimed a "why". For the purpose of my argument, I don't care about the "why". (For other purposes I may. But not this one.) All that is necessary is the "what".

[ForHackernews]

We don't have to play OpenAI's game. Just because they stick a cartoon mask on their algorithm doesn't mean you have to speak into its rubber ears. Surely "hacker" news should understand that users, not designers, decide how to use technology.

LLMs are not people. "Agentic" AIs are not moral agents.

[dirkc]

> a semantic quibble

I mean, all of philosophy can probably be described as such :)

But I reckon this semantic quibble might also be why a lot of people don't buy into the whole idea that LLMs will take over work in any context where agency, identity, motivation, responsibility, accountability, etc plays an important role.