[poszlem]

From the article: "I’ve thought about this a lot over the last few years, and I think the best response is to stop. ML assistance reduces our performance and persistence, and denies us both the muscle memory and deep theory-building that comes with working through a task by hand: the cultivation of what James C. Scott would call metis."

"What do such machines really do? They increase the number of things we can do without thinking. Things we do without thinking-there's the real danger" - Frank Herbert, God Emperor of Dune

[gdulli]

Also Frank Herbert: "Once, men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them."

[chungusamongus]

I mean, people are talking about the butlerian jihad without any sense of irony or subtext. Dune is literally a feudal hellscape that takes place in the wake of that event. It didn’t make things better. Lmao

[yubblegum]

I agree that many people miss the subtle irony of Frank Herbert's books. He seems to be debating himself to a certain extent in that series.

That said, there is no obvious reason to posit that the intergalactic feudal system, CHOAM, or the empire, came to be because of the butlerian jihad. The concrete side effects of the jihad were in fact hyper specialization of cognitive faculties in humans: mentats, guild navigators, and soldiers all possess super human specialized abilities.

[chungusamongus]

Yeah. I'm not saying the jihad was the cause. But elimating AI didn't prevent the feudal system. It didn't really help, in other words. Honestly, I kinda think Herbert just didn't want to have AI or sophisticated robots in his narrative, so he contrived an elaborate reason why that tech doesn't exist.

[yubblegum]

But it did "help". Mentats had supercomputer computation capabilities; navigators folded space and charted non-collision paths; warriors had robocop like abilities. These were developed precisely because the use of thinking machines was forbidden.

I don't think feudalism is going away one way or another. It persists [in various forms] because of certain biological realities, ranging from genetics to loyalty engendered by familial relations. [This is merely an observation.]

In sum, the argument against current AI trends isn't that once addressed we will wake up in utopia. No. The point is that these natural tendencies of humans are hugely amplified and set in generational stone once the elite have control over thinking machines and lord it over a population that has experienced generationally diminished independent cognitive abilities.

p.s. All this somehow reminded me of 'Spock's Brain' episode of Startrek /g Note: the elite there were overcome because Kirk and his landing team were cognitive high performers ..

[keeda]

I suspect the cause-and-effect in creating the narrative is the reverse of what's in the narraitve: Frank Herbert wanted the intricate dynamics of the Guild and Spice and Mentats and exciting close-quarter combat for a more intriguing narrative. But AI and robots made those all obsolete, so he made it disappear with a handwave of "because Butlerian Jihad."

I always thought the Butlerian Jihad was the biggest plot hole in Dune, but I deeply appreciate the world and narrative it enabled.

[chungusamongus]

There are a number of different things being conflated here. My inital statement was just acknowledging the lack of appreciation of the subtext behind the Butlerian Jihad. People are unironically embracing it, which I gather is not really how the event functions in Dune.

At the level of the text, none of those things you mentioned strike me as positive developments. They just siloed computation to a biological track and those biological resources are employed by those in power, which is the same problem in a different form.

This is an aside, but feudalism is not inevitable. The vestiges of it still exist, but capitalism largely upended it.

[nonameiguess]

I think your other comment is basically right that what Herbert really wanted was a fantasy epic with feudal structures, knife fights, and humans with near-magical abilities, and reverse engineered a vaguely plausible future that might bring that about without invoking any actual magic. Arguing whether this is supposed to be considered "good" or not is kind of beside the point. Fiction novels are mostly meant to present worlds that are interesting to read about more than advocating for or against those same conditions replicating in the real world.

The only thing I've really taken from what Herbert himself said, not something a character in one of his books said, is distrust of messiahs and centralized power being an inherently corrupting force, even in the hands of good people.

Unfortunately, I would have to say right now my bets on the most plausible fictional future becoming reality is WALL-E.

[LelouBil]

At least WALL-E had a good ending.

[gdulli]

You think his argument was that we should welcome the likes of Google controlling the direction of our cognition? The book was about the dangers of asserting our independence from those who control technology? Admittedly, I haven't read the sequels.

[chungusamongus]

They didn't assert independence though. That's my point. They just siloed computation to biological organisms, and it led to the concentration of power anyway.

[TeMPOraL]

> "What do such machines really do? They increase the number of things we can do without thinking. Things we do without thinking-there's the real danger" - Frank Herbert, God Emperor of Dune

I always preferred this take:

“Civilization advances by extending the number of important operations which we can perform without thinking of them.” ― Alfred North Whitehead

It's both opposite and complementary to your Frank Herbert quote.

[delecti]

I think it's important that we recognize and understand how those operations are being done, and ignorance of the complexity of all the parts of our lives leads to the death of expertise. People who would learn a lot just from reading the course description of a 100 level class in a field are assuming their lack of knowledge means there's no complexity there.

> “There is a cult of ignorance in the United States, and there has always been. The strain of anti-intellectualism has been a constant thread winding its way through our political and cultural life, nurtured by the false notion that democracy means that 'my ignorance is just as good as your knowledge.” ― Isaac Asimov

The easier society makes it to be unaware of the complexity of everything around us, the easier it becomes to assume everything is actually as simple as their surface-level understanding.

[ori_b]

It's very clear to me that many people have achieved peak civilization -- no evidence of thought remains.

[notpachet]

I guess it hinges on your definition of "civilization".

[hackable_sand]

Or "advance"

[wmeredith]

> ML assistance reduces our performance and persistence, and denies us both the muscle memory and deep theory-building that comes with working through a task by hand

On one hand I intuitively think this is correct, on the other hand these very concerns about technology have been around since the invention of... writing.

Here is an excerpt of Socrates speaking on the written word, as recorded in Plato's dialogue Phaedrus - "For this invention will produce forgetfulness in the minds of those who learn to use it, because they will not practice their memory. Their trust in writing, produced by external characters which are no part of themselves, will discourage the use of their own memory within them. You have invented an elixir not of memory, but of reminding; and you offer your pupils the appearance of wisdom, not true wisdom"

[miltonlost]

And you know, Socrates was right. We did lose our memory with writing! How many phone numbers do you remember now that you have a phonebook in your phone? Humans will lose skills due to LLMs. That's just obvious on its face by the fact that if you don't do a skill regularly, you will lose it (or lose to do it as well as you once had).

[mwigdahl]

The real question is whether we're worse off or better off overall than we were in Socrates' Athens.

[mplanchard]

Regardless of how you feel about this question, it doesn't necessarily map to the current situation.

Just because the loss of one skill to a supplanting technology led to one kind of societal change, does not mean that the loss of any skill to a supplanting technology will lead to the same kind of societal change. Assuming that to be true is a faulty generalization.

I think it wouldn't be hard to argue that writing has changed human society more profoundly than any other invention. Whether or not the change was positive is a matter of taste and likely unanswerable. The point though is there are plenty of other examples of new technologies that changed technology and deskilled humans, both mentally and physically, that changed society in radically different ways, compared to writing (looms, tractors, sails, calculators, computers, guns, and so on).

There's certainly a case to be made that, of major past technological advancements, the kind of deskilling we'd see due to heavy AI use is most comparable to the deskilling due to writing: presumably there were many day-to-day and essential activities that made use of the mental acuity people would lose due to reading, just as there are many day-to-day activities that one can imagine people becoming less skilled in due to AI use.

To me, the most dangerous difference though may be in what gets deskilled. If we only relinquish our ability to do certain menial and intellectual drudgery, that is one thing. But if what we actually relinquish and deskill is our agency and discernment, as a result of constant "delegation" to AI systems, I think we're in for a much worse time.

[randallsquared]

There's a distinction to be made between "worse off" and "worse". Socrates was arguing that writing-users would be worse as people, not that they would experience lives they didn't like as much.

[mwigdahl]

Agreed. And I think he was wrong. Literacy allows individual humans to be exposed to and understand far more of the world's culture and knowledge than the conveyance of knowledge through recitation of epic poems would ever have allowed.

Hell, I would never have had the pleasure of arguing with you without it! :)