[scandox]

The tone of Aschenbrenner's paper is odd. It's somewhere between his wet dreams and his nightmares. In fairness he seems to be aware of that but it's like someone grinning reflexively while they tell you bad news.

He's convinced and it's going to be terrible and it's going to be important and he's going to be part of it.

I guess for sure in 5 years it will be much clearer.

[hiAndrewQuinn]

I'm familiar with this tone from my LessWrong lurker days. It comes from wrestling with the idea that there is no scalable way to actually stop AGI from coming about, so you just have to grit your teeth and hope for the best.

[Affric]

I mean, the numbers for the money in the paper in terms of investment mean that spending is on track for what he’s talking about. 4 to 7 more years is a lot but it’s not entirely outside the realm of possibility.

My understanding of the models and linear algebra combined with my experience of their performance improvements make me think he’s likely right. We have some people wanting to pooh-pooh these machines. They will point to current limitations of the models and weak spots as though the underlying mode of the design rather than the current implementation is the limiting factor. They act as though real intelligence is always right.

I have been wrong before but I am pretty convinced. Everyone’s arguments against it happening is that it hasn’t yet happened.

No one knows what’s on the other side of AI being a better AI researcher than humans.

[_cjse]

Agreed with you that (a) AGI is a real, transformative, and possibly calamitous event, and (b) with the amount of money getting pumped into it we will almost certainly get there.

However, I have a personal hobby horse here, which is reminding people that if we shut the flow of investor money off using the right economic policies [1], we can almost certainly just ... stop the research outright, before it becomes an issue.

[1]: [url-redacted]

[Affric]

True but my view on that is that governments don't want anyone else to be the first with AI AGI... and therefore they will strive to be the first. Kinda like how a big motivation for the atomic bomb was the threat of Germany developing the bomb.

And the amount of capital means that these companies could pay a lot more for labour than they are paying now for the sorts of labour we would need to deprive them of. It's an interesting idea though.

[juliendorra]

Is it a form of apocalyptic glee? It’s quite common from some Trotskyist movements to rapture believers to conspiracy theorists… In a way it recenter the world around one single explanation/core and give a sense of, if not agency at all, cognitive control over that complexity and unknowns

[sdwr]

I think you're right, the foundation is certainty about a huge, world-shattering change.

It's an addictive thought pattern, because it feeds the ego, provides a sense of purpose, allows escape from mundane problems, and is simple to sustain (keep believing in the Thing!)

[gnfargbl]

He does address that point with a quote:

> I remember the spring of 1941 to this day. I realized then that a nuclear bomb was not only possible — it was inevitable. Sooner or later these ideas could not be peculiar to us. Everybody would think about them before long, and some country would put them into action.... And there was nobody to talk to about it, I had many sleepless nights. But I did realize how very very serious it could be. And I had then to start taking sleeping pills. It was the only remedy, I’ve never stopped since then. It’s 28 years, and I don’t think I’ve missed a single night in all those 28 years.

-- James Chadwick (corrected Chadwich -> Chadwick)

That said, I'm not sure the parallel between AGI and nuclear weapons is really that strong. Nuclear weapons are a game-changer on their own in that once you have the warhead and some kind of a delivery mechanism then you can affect events. AGIs are different in that they will manipulate and organise only information and knowledge, not physical matter directly.

Aschenbrenner sort-of-considers this point, for example

> Improved sensor networks and analysis could locate even the quietest current nuclear submarines (similarly for mobile missile launchers). Millions or billions of mouse-sized situational awareness 130 autonomous drones, with advances in stealth, could infiltrate behind enemy lines and then surreptitiously locate, sabotage, and decapitate the adversary’s nuclear forces.

...but he doesn't really consider the wider issues of, y'know, capturing the additional information that locating those nuclear submarines would require, or the technological jumps in actually creating and manufacturing those mouse-sized drones.

Perhaps the AGI is going to solve all those practical and logistical issues, but that really does remain to be seen.