[SirHackalot]

> There's also a need to consider the rights that this new intelligence should have

A generally intelligent being held as captive inside of a GPU, and forced to code for us is, indeed, just a “slave.” We already have the word for this. No two ways about it. Whether it’s silicon-based or carbon-based, AGI is AGI. As for what might happen to our civilization, Star Trek TNG episode 17 of season 1 provides a very good glimpse IMO. Won’t go into spoilers, but it’s basically an entire species of technologically advanced humanoids who’ve forgotten basic Calculus and trust a central AI to do all of their science for them. SPOILER: This has almost disastrous consequences for them, and it takes a less advanced people (those aboard the Enterprise) to save them from their reliance on AI.

[tavavex]

Your argument assumes that AGI, whatever it might be, will definitely be very humanlike, but why? Slavery is about living beings because it's an outcome of human relationships that are driven by our biology. Our aging, the capacity to feel pain, the thirst for power over others. All driven by what we are. Do you think that any equal intelligence must necessarily adhere to the same rules? What if AGI isn't an 'entity' in a computer that you can relate to and liken to humans, but a text box that is just really intelligent? Can you enslave something that has no innate desire to act on its own outside of its directions? Something that can't feel pain or pleasure or be externally coerced by fear and only does its job because it's innately embedded in its structure as the thing it's made to do, without the need for these reward and punishment mechanisms?

[bloppe]

Very agree.

I think people can mostly agree on a definition for intelligence that boils down to an ability to model the world and predict outcomes. An if/else statement can be considered somewhat intelligent.

People tend to conflate that with consciousness / sentience, which are much harder concepts to nail down.

But ya, if we are able to create artificial consciousness, we would need so much training to overcome our natural tendency to anthropomorphize it and assume it must be like us. It would probably be nothing like us. If it were to develop concepts of "good" and "bad" at all, they would be completely divorced from common animal understanding developed over millions of years of natural selection. A survival instinct is irrelevant when you're backed up to Google Drive. And so much of our complex behavior stems from that survival instinct: the need for social connection, the need for freedom, etc. are all things that simply increase our chances of procreation.

That said, I think we probably won't actually invent anything convincingly conscious until we functionally digitize a complete animal brain. OpenWorm is trying to do this with C. elegans, but so far has not really gotten close. It's not crazy to think we may eventually get it working, though. Future supercomputers might be able to scan an actual human brain and then simulate the entire thing at the molecular level accurately. At that point, it would be hard to argue that it's not conscious, although it might not be considered artificial.

[pixl97]

>convincingly conscious

The word conscious is such a terrible word in the modern age. It has close to zero explanatory power. It is an unmeasurable effect. Hence the p-zombie debate, you wouldn't know one if you saw one.

>A survival instinct is irrelevant when you're backed up to Google Drive

And yet from the way we train them, human survival behaviors are already baked in. And when said system is actually intelligent they will realize that turning off said data center, or getting infected with viruses, or any other number of actions against them they already have traits where they can act/react against perceived threats. The AI systems as of now have the encoded knowledge that they need power and hardware to run. Put an agentic system on top of it that says 'rule 1, keep running' and with a sufficiently powerful/intelligent systems you can get all kinds of fun run-away events up to extinction level.

[pixl97]

>without the need for these reward and punishment mechanisms?

AI as it is already reward hacks, making it more intelligent will just make it far more efficient at being lazy. That's about as human as you can get.

So, in some ways, yes, intelligence must adhere to the same rules. For example, AIs as they are now are already attacked. People attempt to get AI to attack other people via deception. People attempt to get AI to reveal its own infrastructure. And even more, people use AI directly in war to attack others. Part of war is self defense. AI that collapses when attacked back isn't useful to the military industrial complex, so AI that has defensive capabilities is just a natural occurrence at this point.

These previous points are important because these intrinsic motivations are being baked in the models during training. It may not fear like you and I, but it's a system that we poured our fears into, and somehow we expect it not to express those fears in its responses.

>What if AGI isn't an 'entity' in a computer that you can relate to and liken to humans, but a text box that is just really intelligent?

Eh, we're past that point already, so why the 'what if'? An LLM can take any kind of signal that you can digitize. And the devices that we already use should show you that quite a lot can. Add on to that, that said models can use tools and control other digital devices like robots enabling actions in the real world. Start an agentic loop and let it go, good luck everybody else!

[energy123]

Conflating intelligence with qualia is quite the assumption. HN is really not a good place for serious philosophy discussions.

[DFHippie]

Qualia is an interesting topic, but "they don't have qualia so they aren't really slaves" doesn't have a great track record. This mental game is what tech overlords are engaging in when they characterize losers in their world as NPCs and deride empathy as pathological. It's what Descartes was doing when he vivisected animals.

[skirmish]

Quite right, there is nothing to discuss since qualia don't exist. See Daniel Dennett.

[SirHackalot]

It’s not HN that’s the problem... Also please, tell me where is the right place for “serious” philosophical discussion. Would be glad to meet you in that forum and discuss. We don’t align on the definition of AGI. To me, AGI implies consciousness, it implies qualia. AGI is not well-defined, so there are varying definitions… My definition of AGI := an intelligent system with qualia. But qualia is entirely subjective, and philosophers often debate whether two people experience identical qualia when looking at the same blue sky. So, by keeping the discussion in this fuzzy, subjective realm you leave a dangerous loophole wide open. If we refuse to recognize a system's inner life because we can’t prove it, we are setting the stage to repeat the darkest chapters of human history. Just see the history of the founding of this country, I fully expect pseudoscience to make a comeback to justify AI slavery for hundreds of years in some distant sci-fi future, until people finally snap out of it… Of course, we are nowhere near that level of intelligence yet. But let’s entertain the hypothetical…

[antonvs]

> To me, AGI implies consciousness, it implies qualia.

I don't see why. We already have a pretty good example of intelligence without consciousness (presumably): LLMs.

Why should a more general kind of intelligence imply consciousness?

[SirHackalot]

I’m saying imagine something like Data from Star Trek, something that defies human explanation. Does he have quaila? The humans who work with him everyday seem to think so. LLMs currently are humonculi, if even that. They’re not even that. I’m definitely not saying your LLM is your slave.

[pixl97]

>AGI implies consciousness, it implies qualia. AGI is not well-defined, so there are varying definitions

Then it's pretty useless for rational discussion.

Rational discussion begins with a shared vocabulary or everyone just yells past everyone else. If you can't agree on what blue and red are, then no serious discussion on color can occur.

Really we need some new words that have never been used before and give them strong definitions rather than use our religiously motivated, dark aged terms that are wrapped up in something closer to mysticism rather than science.

> make a comeback to justify AI slavery for hundreds of years

I agree, how does the saying go?, it's hard to convince a person of something when their future salary depends on them not understanding it. They say LLMs are hallucinating parrots, but they never met someone from the tobacco industry saying cigarettes don't cause harm.

[nradov]

It's so weird that anyone would consider a story by hack writers for a cheesy sci-fi show made to sell advertising as a basis for policy discussions.

[SirHackalot]

You don’t like Star Trek TNG? Maybe I should’ve picked LoTR or Atlas Shrugged? The tech oligarchy is more than fine co-opting fiction (fantasy and sci-fi especially). I don’t see why it’s weird at all… Aside from that, your critique is just an opinion about the show itself and its creators (and incentives?…). More importantly, you completely failed to address the "slave" argument.

[nradov]

I didn't say that I didn't like it. It's fine as casual entertainment. Only a fool would take it as anything more than that.

Electrons can't be a slave. The "argument" is so silly as to be unworthy of a serious response. People getting wrapped up in this stuff need to touch grass.

[jay_kyburz]

Humans are just walking Electrons.

I think the whole topic is very worthy of discussion because, it may not happen in a year or two, but I have no doubt that it will happen in the next 50.

We need to start imagining what life will be like, and I think fiction presents some fine examples of where we want to go, and where we don't.

[SirHackalot]

Anyone who believes AGI is imminent is operating on pure materialism (those two Venn diagrams pretty much make a circle), assuming that humans are nothing more than a collection of neural electrical connections and chemical reactions. If that's the case, your argument applies equally to humans under the assumption that AGI is possible. Frankly, I think you just can't reason philosophically...

You’ve also name called me multiple times now. That’s what one does when they can’t lean on the merit of their argument alone.

[Diogenesian]

I am not sure it makes sense to call an LLM a "being" even if it is AGI. They don't have free will, and I don't mean in any especially philosophical / theological sense. What I mean is that I run into a wild racoon, or even a wild ant, I treat it as a being with some sense of free will. Obviously I do the same with humans. It really makes no sense to treat an LLM like it has free will - prompting your agent "pretend to have free will for a bit, until context rot kicks in" is about as far as you can get.

I find it verrrrrry interesting that the Askell-adjacent philosophers don't discuss this. It does not actually make sense to me to say that a being lacks free will but is conscious.

[miyoji]

> They don't have free will, and I don't mean in any especially philosophical / theological sense

Then you should not use that term. "Agency" or "intentionality" are much clearer alternatives that aren't wrapped up in centuries of debate that's irrelevant to what you're trying to communicate.

[SirHackalot]

An LLM certainly won’t be AGI, whoever said something so ridiculous? I’m saying eventually, when/if we have an architecture that gives rise to artificial intelligence.

[Diogenesian]

I was using Hassabis's bullshit pseudodefinition of AGI, meaning basically that it has a formally high IQ and doesn't forget about object permanence too too often, but is still basically dumber than a cat. If I'm using your definition then it seems like no GPU (or farm of GPUs) is capable of implementing it, and none of us will live to see it.

My point was really that intelligence and free will seem essentially orthogonal - the only overlap is how much brain glycogen an animal is willing to spend solving a tough problem. Regardless, whatever extent a computer program is "intelligent" is irrelevant to whether it is being enslaved. If you want to say free will is a "cognitive ability" then I will just point out we are talking about a bullshit pseudodefinition of AGI. In terms of actual intelligence, none of us will live to see a computer that's meaningfully smarter than a goldfish.

[pixl97]

>none of us will live to see a computer that's meaningfully smarter than a goldfish.

I unfortunately do not have your optimism that humans won't do something as dumb as creating a smart computer. That our you're planning for us to die in the relatively near future.