[diego_moita]

I think the best take in AGI is Edsger Dijkstra's:

    “The question of whether a computer can think is no more interesting than the question of whether a submarine can swim.”

I am not interested in computers that have their own intelligence but I do want computers that increase my own intelligence.

[grumbelbart2]

I've read this multiple times and disagree.

If I had an AGI that designs me a safe, small and cheap fusion reactor, of course I would be interested in that.

My intelligence is intrinsically limited by my biology. The only way to really scale it up is to wire stuff into my brain, and I'd prefer an AGI over that every day.

[diego_moita]

I get the idea.

But if I can't understand how and why the fusion reactor is safe, small and cheap, I wouldn't consider it safe.

Very much the same way that I don't take Claude Code's changes to my code without understanding what it does.

Augmenting my intelligence is non-negotiable. I want to be in control.

It is the classic "trust but verify". And I need my own intelligence to verify.

[ACCount37]

What is the value of you in this system?

If an AI is advanced enough to design mass production ready fusion reactors, I can't help but feel that the "human in charge of that AI" quickly becomes fully redundant. There is nothing that human can do that the AI itself can't do faster, better and cheaper.

[diego_moita]

True, mostly.

This idea of "AI taking control" reminds me a lot what Karl Marx would say about "capital".

He wrote his things during the blooming of the second industrial revolution, a time when machines were replacing humans and forcing forward new economic, social, political, cultural and labour relations. And a key issue he stressed a lot is that this diffusion of machines and capital reshaping society was brought forward by a class of people that he called the bourgeoisie. He stressed a lot that it was a power struggle within society.

We're going through something similar today with the information technologies reshaping social relations. And the Bezos/Zuckerberg/Altman/Ellison of today are similar to the industrial barons from the Gilded Age. But, the same way that people reacted against the full-blown wild capitalism from 19th century's second half, we might also see some reactions against the advance of this techno-plutocracy.

In particular, I am optimistic about how the EU and some 3rd World countries (e.g. India, Brazil) are placing restrictions on social networks and techno-cartels.

> What is the value of you in this system?

So, to answer your question: individually I can't go beyond much more than careful choices (avoid cookies, stay out of Facebook, etc). Collectively we can make political choices. Ultimately, the most consequential political choice is move away from countries that give all power to the techno barons.

[imiric]

Whether or not that question is interesting is hardly relevant in this particular discussion. Companies building this technology are marketing it as if computers can think. They're telling us they can, and people are buying into it based on that claim.

So that quote doesn't reconcile the extraordinary claims of one side with the skepticism of the other.

And also, no disrespect to Dijkstra, but that sentiment is a bit shortsighted. If we could make computers think, it would have a profound impact on humanity. This is why there is so much excitement around this. We've been imagining this scenario for centuries, and we hope that this time around we can finally crack it. So comparing that achievement with something we can produce with classical technology is... uninspiring? Underwhelming? Selling ourselves short? I can't quite put it into words, but the possibility of answering that question would certainly be very interesting.

[diego_moita]

> and people are buying into it based on that claim.

I say this remains to be seen. You know that a lot of times you see the expression "AI" in the news. it comes followed by the word "bubble", right? If we see a big crash on the AI companies stocks we'll have proof that people aren't buying. And I strongly believe we'll see this crash and I think smart people aren't buying it.

OTOH, I think we need to be careful with the usage of the word "think". Dijkstra would probably give it a very broad meaning, going from French Impressionism, Bach and Shakespeare to Relativity Theory, Evolution Theory or Quantum physics, maybe even to Maradona's or Johan Cruyft's feet (Dijkstra was Dutch, remember). Computers and AI might go very deep in their "think" but will be very, very bad at the broad game. Frankly, I don't see how Markov Chain based technologies (e.g LLMs and most of AI today) can stop being replicators and start being innovators.

It is a bit like Pablo Picasso's quotation: "Computers are useless, they can only provide us answers".

[quantum_state]

Well said!