I'm a big fan of Scott's, but I haven't been happy to see how this story's failure to say what it means has in some ways only made it less relevant over the years as we've come closer to building something like the item that, on the surface, it talks about.
Not to say I think it's a bad thing at all that people have a name for this concern for giving up agency to an external system, even when that system really has your best intent in mind. You can call Claude a whispering earring and people will understand what you mean pretty well. The actual consequences you're looking out for are maybe a bit subtler than your brain atrophying, I really really never liked that detail in the story, but that's fine, people will get it. By the Death of the Author, the story's a great contribution as it is.
But it just feels like a loss that there really was a different thing that the author meant, and it got lost in translation. http://web.archive.org/web/20121007235422/http://squid314.li... clarifies it. It's quite similar to a Yudkowskyan fun theory concern (and IMO really not surprising that the story was released a couple of years after the fun theory sequence): If making the right choices and doing the right thing eventually comes down to a mechanical computation, and you've got all of your resource limitations and psychological hang-ups and such things out of the way, then there's no room left to do the right thing, and you've turned yourself into a mechanism.
I don't think that's really a possible state that any system with any trace of humanity left in it will ever enter, there's too many too large search processes in the way to fit in a finite universe. So it's a pretty abstract concern even among its peers. But it's one that's fun to talk about sometimes, and one lesson we've learned from the response to this story is that mysterious, easily misinterpreted parallels in fact do not point people in the right direction to see it.
It's not subtle, it's clearly saying "if you outsource your thinking to something else, your mind will wither".
People complaining about this have a long and storied history, this is the same as Plato complaining philosophers were ruining their minds by writing things down to remember them.
This is not what Plato thought, this is not even what Plato wrote that Socrates thought. You are referring to the Phaedrus. You should read it! Its much better than this story, trust me.
But either way, people's minds do not wither in this story, it explicitly works out fine for them?
> But either way, people's minds do not wither in this story, it explicitly works out fine for them?
Huh?
"when the corpses of the wearers were being prepared for burial, it was noted that their brains were curiously deformed: the neocortexes had wasted away, and the bulk of their mass was an abnormally hypertrophied mid- and lower-brain, especially the parts associated with reflexive action"
You could infer their minds fading away even without that paragraph, but it's very explicit.
Also you have another comment where you insist nobody has explained what the story is arguing when multiple people told you what the story is arguing, what the heck.
OK, so there is a magical device that provides you with deep contentment no matter what. But then, when you die and they cut your head open, the physical, biological brain has shrunk. And that's bad. Is that right? Is this what it's about?
Is the story actually arguing something? I have just been literally asking what it even presumes itself to be about. Is it the gloss above?
If it's about agency, I don't get it. It seems silly/facile to assert the "right" decision is the thing that matters without making the thing so magic it doesn't even make sense. Does agency stop you from getting hit by a runaway bus? Help you from being persecuted for arbitrary reasons? The story tells us they all, unfailingly, live "abnormally successful" lives. That's all supposed to be because of their (non) agency because of the earring device?
Is it actually contentment or are you fading away so much you cease to exist?
The story is not about physical brain changes, no. It's about losing yourself. It's arguing that to exist is to make decisions. Following the device too much leads to a life that looks successful from the outside but inside no longer has a person enjoying the success, just a puppet.
Eh I think the writer should visit poor countries and live among locals. Mistakes and poor judgements compound, and it leads to horrific living conditions (malnutrition, nonexistent hygiene, quack medicines, short-sighted violence, etc)
In fact developed countries are built on top of these "earrings": it's called math (and by extension science in general). If the math is right, it's always right, and it has spared their lucky citizens a LOT of needless misery and wheel-reinvention.
"Thinking for yourself" is overrated if you don't have good framework. See antivaxxers and their "research"
No, this story takes it to the extreme, that you end up basically having no brain.
Take it this way. People seem to judge that LLMs are not conscious and thus have no real inner experience. And that humans do. So if you replaced a human brain with a very powerful LLM (with some simply residual neural matter), you've basically killed the human. He isn't feeling anything at all anymore, he's just an automaton, basically a flesh robot.
I think this story is a weirdly good discussion point because many people still don't seem to "get it" that success is completely fundamentally irrelevant, and what matters is having a good inner life. You are what happens in your mind -- overall good thoughts and inner life = good life; bad thoughts and inner life = bad life. You can theoretically be the richest man in the world and about the most miserable, and very poor and living the best life.
If we don't kill our definition of good life and success (i.e. reaffirming value as pertaining exclusively to the mind), it will sooner or later kill us, or kill what's good in us. The more we can orient our entire society toward good inner lives and not anything else, the better our lives will be.
Then went back to a society safeguarded by the "earrings" (OSHA, FDA, CDC, etc). Poor countries usually don't have the resources and enough HDI to enforce public safety.
Not saying his experience doesnt count, but 1 week is not enough to experience that crushing learned helplessness when everyone around you keep doing stupid sh*t and sabotaging themselves and others near them.
Which is why Trump defunding public safety agencies is baffling to me, a person from a poor country. Such a decadent behavior.
Not to say I think it's a bad thing at all that people have a name for this concern for giving up agency to an external system, even when that system really has your best intent in mind. You can call Claude a whispering earring and people will understand what you mean pretty well. The actual consequences you're looking out for are maybe a bit subtler than your brain atrophying, I really really never liked that detail in the story, but that's fine, people will get it. By the Death of the Author, the story's a great contribution as it is.
But it just feels like a loss that there really was a different thing that the author meant, and it got lost in translation. http://web.archive.org/web/20121007235422/http://squid314.li... clarifies it. It's quite similar to a Yudkowskyan fun theory concern (and IMO really not surprising that the story was released a couple of years after the fun theory sequence): If making the right choices and doing the right thing eventually comes down to a mechanical computation, and you've got all of your resource limitations and psychological hang-ups and such things out of the way, then there's no room left to do the right thing, and you've turned yourself into a mechanism.
I don't think that's really a possible state that any system with any trace of humanity left in it will ever enter, there's too many too large search processes in the way to fit in a finite universe. So it's a pretty abstract concern even among its peers. But it's one that's fun to talk about sometimes, and one lesson we've learned from the response to this story is that mysterious, easily misinterpreted parallels in fact do not point people in the right direction to see it.