[satori99]

> You might want to live there, but I wouldn't. Virtually all humans in the books [...] are kept as pets by the ships, for amusement, basically as clowns.

I got the impression that the Minds are proud of how many humans choose to live in their GSV or Orbital, when they are free to live anywhere and they appear to care deeply about humans in general and often individuals too.

Also, the Minds are not perfect Gods. They have god-like faculties, but they are deliberately created as flawed imperfect beings.

One novel (Consider Phlebas?) explained that The Culture can create perfect Minds, but they tend to be born and then instantly sublime away to more interesting dimensions.

[Vecr]

> One novel explained that The Culture can create perfect Minds, but they tend to be born and then instantly sublime away to more interesting dimensions.

That shouldn't happen. No way would I trust an AI that claims to be super, but can't solve pretty basic GOFAI + plausible reasoning AI alignment. In theory a 1980s/1990s/old Lesswrong style AI of a mere few exabytes of immutable code should do exactly what the mind creating it should want.

[impossiblefork]

To some degree the point of the culture novels is that AI alignment is just wrong, imposing things on intelligent beings.

The civilisations in Banks stories that align their AIs are the bad guys.

[Vecr]

I guess? That's not really a possible choice (in the logical sense of possible) though. "Choosing not to choose" is a choice and is total cope. An ASI designing a new AI would either have a good idea of the result or would be doing something hilariously stupid.

I don't think the Minds would be willing to actually not know the result, despite what they probably claim.

[impossiblefork]

It is actually a choice that we do have.

We could easily build AIs that just model the world, without really trying to make them do stuff, or have particular inclinations. We could approach AI as a very pure thing, to just try to find patterns in the world without any regard to anything. A purely abstract endeavour, but one which still leads to powerful models.

I personally believe that this is preferable, because I think humans in control of AI is what has the potential to be dangerous.

[Vecr]

The problem is that some guy 24 years ago figured out an algorithm that attaches to such an AI and makes it take over the world. Maybe it's preferable in the abstract, but the temptation of having a money printing machine right there and not being able to turn it on...

[satori99]

A Culture Mind would be deeply offended if you called it "An AI" to its avatars face :P

[Vecr]

A few exabytes is enough for a very high quality avatar. Maybe the minds are funny about it, but the option's there if they want them to stop leaving the universe.

Remember that "a few exabytes" refers to the immutable code. It has way more storage for data, because it's an old-school Lesswrong style AI.

Not like a neural network or an LLM. Sure, we dead-ended on those, but an ASI should be able to write one.

> A Culture Mind would be deeply offended if you called it "An AI" to its avatars face :P

That's how they get you to let them out of the AI box.