[throwaway115]

>What if someone builds a supermassive teapot in Earth's orbit and it breaks free and comes hurtling towards us?

Ok, that seems ridiculous and infeasible and nothing that has happened so far indicates that this is a real threat.

>Yes, but it would be so catastrophic to humanity, that we must take it seriously, regardless of how improbable it is!

This is the AGI debate, in my view. If we're picking different extremely improbable events to get worked up about, why stop at AI?

[circuit10]

It’s arguable whether it’s actually improbable. Many people, including (and seemingly especially) experts think it’s likely or even inevitable. This video might be interesting to see some arguments for why people think it’s important: https://youtu.be/9i1WlcCudpU

[throwaway115]

I know someone who fancies himself an expert, and indeed, by comparison to the average tech worker, he is an expert in LLMs and AI. He vehemently believes AGI is coming this year. At least, that's what he said last year. I suppose he will probably feel sheepish if I bring it up now that we're at month 6.

My point is that the only thing that seems to convince the pro-AGI crowd is having them make specific predictions, waiting until the deadline, then asking them why they didn't happen yet.

[JKCalhoun]

I feel like we'll know right away though. We can then act right away.

I'm not sure what you would do now that is different because you are anticipating such a thing.

[circuit10]

It would be far too late by that point, as it can already take measures to stop us. This is a difficult problem and we really should have started putting a lot more effort into this a lot earlier. Of course we can’t test a superintelligence directly yet, but we should do as much theoretical work as we can

Actually the video addresses this point, the comparison it makes is waiting until we’re already on Mars before we start thinking about spacesuits and airlocks

[Aloisius]

Inevitable? Sure assuming we don't extinct first. In the near future? No.

This is just another in a long line of technology panics. Unfortunately, there always seems to be some "concerned" experts that are both overly optimistic about the speed of technological progress and overly pessimistic about where that progress will lead adding fuel to the fire.

Eventually some other new technology will incite a new panic and this one will become another footnote in history like the fears over grey goo or genetically engineered superhumans.

[circuit10]

What about when people pointed out problems like leaded fuel and climate change and we didn’t take action for far too long? We shouldn’t dismiss it just because of things it sounds similar to. In fact you could always dismiss any concern that could lead to human extinction like that, because if there was an existing example of human extinction then we wouldn’t be around to talk about it; that doesn’t mean it can’t happen.

[agucova]

Agreeing with circuit10's comments, I don't think many proponents of AI Safety are doing so through Pascal wagers. People differ a lot in their assessment of how likely certain risks are, but people I know working on AI Safety tend to place numbers between 1% to 30% of a global catastrophe associated with AI in the next 20 years [1]. I don't think this is negligible, nor analogous to your supermassive teapot example.

[1]: You might disagree with this assessment, but my point is that this is not where most people are arguing from.

[richardw]

Because an increasing amount of capital and effort is going towards improving the chances we eventually achieve AGI. We might hit another winter where the party stops, but right now a lot of stock prices are encouraging more research. That increases the odds. Assuming we never get there is a bet against human ingenuity.

[hehdhdjehehegwv]

It’s not all or nothing. I’d be surprised if we don’t see autonomous quad copters in eastern Ukraine dropping bombs on soldiers by end of ‘25.

The parts and models are off the shelf now, it just needs the first person desperate enough to release a swarm of killcopters with cheap ML classifiers to make it a reality.

Curious how people will feel after the first autonomous swarm becomes widely known, because it will be replicated widely.

[Angostura]

Is there a company out there building large orbital teapots? If so, you analogy is a good one. But I haven't heard of said company