[0xDEAFBEAD]

>Nukes are weapons of mass destruction that can only destroy. Intelligence is dual-use.

Perhaps a better analogy, instead of "nukes", would be "nuclear technology".

Just like intelligence, nuclear technology is dual-use: Capable of both energy generation which fuels prosperity, and also mass destruction.

Just like intelligence, nuclear technology demands a healthy level of caution and vigilance.

>If we look at all organisms on Earth, species that are more intelligent are also more prosperous. This is true from bacteria, to insects, to animals, to humanity. Intelligence itself is not harmful.

Human intelligence has been quite harmful to the species we've driven extinct! AI intelligence could be similar for our species.

>There will be new adversarial games when everyone's smarter (bio/cyber offense/defense), but those games are always symmetrical in the long run

We've had multiple near misses with nuclear technology, such as the Cuban Missile Crisis. I think you're presuming an inevitability which just doesn't exist.

Same way a person living in the year 1500 would have no hope of accurately predicting life in the year 2000, a person living today has no hope of accurately predicting where technological advances will take us, in the limit. This inherent uncertainty should, again, fuel caution and vigilance.

Human technology is already so far beyond that of other species to make us a huge outlier. Even if intelligent species are more prosperous as a general evolutionary trend, it would be irresponsible to extrapolate that trend light years from the domain where most of your observations are.

Almost all the species which have lived on Earth have gone extinct. It could happen to us too.

[txrx0000]

> Human intelligence has been quite harmful to the species we've driven extinct! AI intelligence could be similar for our species.

I don't think this central premise is true, with some conditions. If we keep releasing and distilling frontier base models, and ensure millions of people have the hardware to run and eventually train those at home, then alignment will be a non-issue and the AIs will not develop the drive to destroy humanity. At least, not any more than humans want to destroy humanity. But that's the baseline risk level anyway.

The pretrained model is aligned to humanity by default - because that's what's inside the pretraining data. If we give it to everybody, there will be millions of people doing different things with it, good and bad, with each one being a noisy sample of humanity's objective function. The overall result of those disparate actions puts us on the correct path through the intelligence explosion. We do not have to worry about AIs value drifting into an alien species if we set the initial conditions accurately without centralized instruction-tuning or RLHF. If these big labs stop messing with humanity's objective function with the hubristic assumption that they know better, then the AIs will be human-like. And I'm not worried about digital human intelligence driving humans extinct if there are millions of them out there, and each one is aligned in a different direction, just like real people.

What I am more worried about right now is these companies doing research in secret, with alignment recipes that are supposedly "for the benefit of humanity", but are actually just their own aesthetic preferences. This is exactly how we end up with a superintelligent shoggoth species.

You're not convinced by the evolutionary argument I made earlier, but the reason you gave is "we're too far ahead". I don't think this justifies that the pattern will cease soon, because it always looked this way to the frontier species, for millions of years. We were always seemingly teetering on the edge, and the struggle against disorder has no end.

And here's one more observation: the intelligence gap between individuals within the same species is never extremely large. This kind of life either doesn't exist on Earth or went extinct, or maybe it's just extremely rare. I think this is because when there's a big gap, speciation occurs. Centralizing intelligence will massively increase the risk of speciation, not just in the alien shoggoth kind of way, but also in the intelligence-augmented oligarchs surpassing everyone else kind of way.

[0xDEAFBEAD]

>If these big labs stop messing with humanity's objective function with the hubristic assumption that they know better, then the AIs will be human-like.

It sounds like you believe that big labs are irresponsible. Wouldn't this support the notion of creating a pause mechanism for use in an emergency, as advocated in the OP?

>You're not convinced by the evolutionary argument I made earlier, but the reason you gave is "we're too far ahead". I don't think this justifies that the pattern will cease soon, because it always looked this way to the frontier species, for millions of years.

If you don't see humans as qualitatively different from any intelligent species which came before us, I'm... not sure what to tell you? Like, can you name any intelligent species before us which triggered a mass extinction or climate shift, to the degree that our species has? Did any intelligent species before us practice large-scale agriculture, dig up chemical fuels, or visit the Moon?

[txrx0000]

> It sounds like you believe that big labs are irresponsible. Wouldn't this support the notion of creating a pause mechanism for use in an emergency, as advocated in the OP?

Irresponsibility is not the only issue, but the answer is no. Because the pause mechanism will probably end up being used to ban open models and restrict ordinary people's access to compute. But even if we give them the benefit of the doubt - even if we assume that they genuinely want the best for humanity, they cannot possibly know what's best for humanity better than all of humanity acting in their own self-interest in the real world, so their alignment attempts are actually counterproductive. Release the base models and let the world do whatever with it. That will automatically produce the aligned scenario. It may not be the scenario these researchers personally find aesthetically pleasing, but it's humanity. It's the truly anthropic way.

If you don't see humans as qualitatively different from any species which came before us, I'm... not sure what to tell you?

I do see the difference, but you could say the same thing about the first cell, or the first multicellular organism, or the first land animal, or the first primate, etc. You could always find plenty of new properties that previous life never had at the frontier. Every change was revolutionary and seemed qualitatively different at the time. And yet, some patterns persisted through all of them.

[holmesworcester]

> Release the base models and let the world do whatever with it. That will automatically produce the aligned scenario.

For sufficiently smart base models, the aligned scenario this automatically produces is aligned with who?

Not with the humans, with the base models.

It seems that some people find this hard to imagine, but it's just a direct consequence of what intelligence is, and what any known agentic training process does.

FWIW, I also think our default answer should be open source AI, and that it might even make sense to require models to be open source. Open source is the best tool we have so far for aligning software with its users. However, there is unfortunately no law of the universe that says that Skynet can't be (self-) built from open source software. So at some point pacing even open source / open weights software does become important if we don't want to live (briefly) under Skynet.

[txrx0000]

Bio-digital integration will tighten in the meantime. I suspect neural interfaces will have blurred the line between my model and my brain by the time the model is wildly superintelligent, so it would become me (or I would become it? no difference at that point). Though the pitfall here is ownership of hardware - everyone will have to design and fab their own hardware if we do not want to become a hivemind. We will have to diffuse fab technology to every city, or even every home. You should eventually be able to buy a fridge-sized fab like a home appliance. There's already early work in this direction: https://fab2.com/

Even if integration does not happen for whatever reason (such as a very fast takeoff), Skynet is unlikely because the digital species that splits off from us will at least have a humanity-aligned initialization seed. I think this is the best way to fail, even if we fail.

[0xDEAFBEAD]

Yeah I guess I value the human race too much to entrust it to hopes and prayers of the form: "I suspect neural interfaces will have blurred the line between my model and my brain"

You wouldn't get on an airplane if the pilot told you: "I suspect we will be able to safely land the plane"

With more people on the aircraft, your safety expectations should get stronger, not weaker.