[bjackman]

I think his agenda here is to point out that your probability distribution for AI outcomes should be broad (what you said), but most importantly: this means you must take seriously the possibility that we are gonna get superintelligence quite soon.

Basically a lot of people say "but isn't it also pretty likely that we DON'T get superintelligence?" And, yes, it is. But superintelligence being even a remotely plausible outcome is a big fucking deal. Your investment choices in that context are not important.

People really struggle to think rationally in the face of this shape of uncertainty.

[torginus]

You want to go to the store to get ice cream. Ice cream is delicious and the value of eating ice cream is a small positive, let's say x. There's a one in ten million chance you'll get hit by a car on the way and die, and your life is infinitely precious, therefore the expected value of going is x times 1 = x, and the one of not going is 1/10m times negative infinity which is negative infinity. You are a rational person, so you don't go. In fact you don't do much of anything. Your value model of every activity has collapsed to a single value.

That's the problem with 'singularity' arguments. The people making them ignore the fact that the mathematical definition of the word means 'the model of outcomes collapses to a single value' therefore the model stops being useful, yet they somehow claim to be able to make predictions beyond the singularity. It's like those shitty Facebook math posts where they divide both sides of the equation by 0 (the fact hidden by some sleight of hand), to 'prove' that 2=1.

The formulation of the singularity involves putting outrageous values into the parameters of the model of reality, and denominator ignorance, and then claiming 'rationally' determining that the consequences are too severe to ignore.

[Balgair]

Aside:

The singularity framing is really tough here, right? It comes from black hole physics. Essentially, at the event horizon, the way we know how to do physics stops working, and we rightly conclude that we can't currently say anything about the other side of the event horizon. It is not saying that nothing is occurring there. Matter, time, space, energy, whatever, that still is there (maaaaybe?) and is still undergoing something. It's just that we don't know what that is.

The same is true with using these tech singularity arguments. Like, in the age of superintelligence (if that happens), there will still be thing happening, the dawn will still come every day and the dusk will still too. It's just that we say our current ideas about that new day aren't that applicable to that new age (God, this sounds like a hippie).

However, unlike with black hole physics where we aren't even sure time can exist like we know, we are likely all going to be there in that new superintelligence age. We're still going to be making coffee and remembering bad cartoons from our youth. Like, the analogy to black hole physics breaks down here and maybe does a disservice to us. It's not a stark boundary at the Schwartzchild radius, it is a continuous thing, a messy thing, a volatile thing, and very importantly for the HN userbase, a thing that we control and have the choice to participate in.

We are not passively falling into the AGI world like the gnawing grinding gravity of a black hole.

[muvlon]

I don't know, are we actually going to be making coffee in the case of the AGI singularity?

If you listen to the hardcore doomers, the misaligned superintelligence will curl a finger on its monkey paw and turn the planet into paperclips or something. If you listen to the most depraved boosters, AGI will remove the need for 99.999% of human workers and so we all get turned into biofuel to churn out more tokens.

Yes those are really extremely scenarios but that's how I think of the singularity. It's so alien that we cannot rule out anything.

[_alternator_]

The event horizon = singularity metaphor is a little off. There is no breakdown in the laws of physics at the event horizon. It's just that there is no light or matter that escapes from the event horizon. But the laws of physics don't break down until you reach the center of the black hole (which will happen in finite time after you cross the event horizon).

So there are a couple interesting and meaningful changes at the event horizon, but it's not a mathematical singularity.

[uselessTA]

There's an objection here when you get to tiny numbers, but surely you wouldn't get ice cream if there was a 10+% chance to get hit by a car?

I think he's saying that >1% or even >10% chunk of your probability mass should be on superintelligence, otherwise you're implicily >99% confident of stalled progress, which seems overconfident. We're not talking about not some infinitesimal fraction here.

[metalcrow]

And yet sometimes the consequences ARE too severe to ignore. Nuclear war is a serious concept and it's carefully investigated and attempted to be controlled by a lot of powerful people. Why is this situation different? Because it's unlikely? So is nuclear war.

[coldtea]

>but most importantly: this means you must take seriously the possibility that we are gonna get superintelligence quite soon.

So, his point with all the demand for rigor is to end on a hand-waved jump of faith from "improved AI models" to the mythical "superintelligence"?

[XorNot]

That's literally the singularity though - the point past which predictions are meaningless.

My "plan" is hope for a benevolent intelligence that establishes a post-human government and then enjoy poat-scarcity society doing wood working or something.

Billionaires should probably be more worried.

[coldtea]

Soylent Green and Mad Max with billionaires in private communities and bunkers sounds more likely than "paid to exist and woodwork"

[anon7725]

On our present trajectory, any new super intelligence will be the billionaire’s plaything. Hope is indeed needed to see how it would benefit the common person.

[XorNot]

I mean I could structure an argument as to why that might be unlikely but that's exactly the point: it's all speculation. We don't know what super intelligence would do. It's meaningless to try and plan for.

[topherhunt]

I think his agenda / point is that, viewed from Lindy's Law, given the SOTA in 2026, superintelligent AI arriving soon is vastly more probable than not, right? To make the case that "sure, AI capability and intelligence have grown exponentially over the past several years, but don't worry, they're about to abruptly level off and in fact won't blatantly surpass human-level intelligence within the coming decades" seems to have a high burden of proof unless your model is less "sigmoid" and more "abrupt plateau".

[coldtea]

>I think his agenda / point is that, viewed from Lindy's Law, given the SOTA in 2026, superintelligent AI arriving soon is vastly more probable than not, right

Why would that be? Nothing about Lindy's Law makes that promise. And even the SOTA in 2026 is over-estimated thanks to a trillion dollar industry trusted to not influence benchmarks.

[stego-tech]

You’re 100% correct, which is why I opted for a broad investment approach rather than trying to pick “winners”.

My thought process RE: superintelligence/AGI is generally this:

* I personally don’t believe it’s likely to happen with silicon-based computing due to the immense power and resource costs involved just to get to where we are now; hence why I invest broadly to capitalize on what gains we actually attain using this current branch of AI research across all possible sectors and exposure rates

* If we do achieve AGI using silicon-based computing, its limited scale (requiring vast amounts of compute only deliverable via city-scale data centers) will limit its broader utility until more optimizations can be achieved or a superior compute platform delivered that improves access and dramatically lowers cost; again, investing broadly covers a general uplift rather than hoping for a specific winner

* If AGI is achieved, nobody - doomer or booster alike - will know what comes next other than complete and total destruction of existing societal structures or institutions. The stock market won’t explode with growth so much as immediately collapse from the disintegration of the consumptive base as a result of AGI quite literally annihilating a planet’s worth of jobs and associated business transactions. In this case, a broad spread protects me from harm by spreading the risk around; AGI will annihilate the market globally, but not all at once barring a significant global catastrophe instigated by it

* Which brings me to the worst outcome, where AGI follows the “if anybody builds it everyone dies” thought process: investment is irrelevant because we’re all fucked anyway.

And that’s just my investment approach. I’m too pragmatic to believe we’re at the bottom of the sigmoid curve, but too wise to begin guessing where we actually exist on it at present or how much is left in the current LLM-arm of AI research; I’m an IT dinosaur, not an AI scientist.

What I can point to is the continued demand destruction of consumer compute through higher costs and limited availability due to rampant AI speculation as proof that the harm is already here in a manner most weren’t predicting, while at the same time actual job displacement by AI is limited to the empty boasting of executives using it as a smoke screen for layoffs after RTO mandates failed to thin headcount sufficiently.

In the USA in particular, we’re facing a perfect storm of:

* consumer confidence collapse leading to a decline in spending on all goods, especially luxury ones, by all but the most monied demographics

* data center-driven cost increases (energy) and resource destruction (land, water, fossil fuel use)

* the eradication of government support for renewable energy that would’ve kept these costs in check

* the widening wealth gaps creating a new underclass not seen since before WW2

In other words, most of the discourse continues to revolve around hypotheticals of tomorrow rather than realities of today. That would be the lesson I’d hope more people take away from something like this, so we can finally begin addressing issues themselves rather than empty online circle jerking about who is right or wrong.

[naasking]

I think this is a bit too pessimistic. Progress in algorithms has matched or exceeded progress in hardware, so the same number of FLOPS spent training GPT-3 years ago would produce a much better model today. Ditto for energy use, and hardware is more efficient at delivering FLOPS.

> the widening wealth gaps creating a new underclass not seen since before WW2

I go back and forth on this. I think the reality is that "underclass" is a moving target. AI and automation makes things so cheap that today's underclass lives better than kings ever did.

[elektronika]

> AI and automation makes things so cheap that today's underclass lives better than kings ever did

I suggest you go share this opinion with the people living on the street because they can't afford housing.

[naasking]

That's a tiny minority. The inconvenient truth is that the vast majority of those living on the street are mentally ill or drug addicts.

[coldtea]

>we’re facing a perfect storm of

Add

Total collapse in government quality AND public trust to politicians

Total collapse of news media to slop and paid-for

Total collapse of culture

(Not just the US either)