[evrydayhustling]

This is a straw man argument. Cowen does not argue against the idea of existential risk, he argues that specifically nobody should believe they can anticipate the consequences of technological change.

You can agree with him and still be existentially concerned about a specific asteroid, or climate change. And you could use a little bit of energy monitoring the unintended consequences of AI, the way we monitor asteroids. But to wish to halt AI advancement requires an unhealthy mix of pessimism and overconfidence in your predictive powers.

[blagie]

The number of potentially world-ending technologies is growing exponentially: Chemical, biological, nuclear, etc.

I don't think any of them individually are high-risk, but put together, I increasingly believe we need to fundamentally change how we govern humanity to mitigate existential risks.

That risk keeps growing as you extend timeline.

I don't know whether we can or should halt AI advancement, but I do believe it's not something which should be market-driven. If we set up free markets, market forces become indomitable. You can't stop them.

[BurningFrog]

> The number of potentially world-ending technologies is growing exponentially

This is an argument for increasing the number of worlds.

[jeremyjh]

That isn't a bad idea but it really isn't an answer that helps us in the next several hundred years. A technological civilization that is fully self-sufficient would need millions of people in every profession like we have on earth. Vernor Vinge estimated 100 million people are required to run a civilization that could refit a starship.

[AstralStorm]

Sufficiently automated and focused, I'd drop one order of magnitude from it. (So 10 million, based on manufacturing, refining and mining estimates, assuming the materials are available etc.) Think of it like applying the city of Shenzhen to the task, if they wanted they could do it. Heck, build one from scratch if materials are there.

Having a (reasonably) sustainable 10 million sized habitat is quite hard.

[jeremyjh]

Shenzen doesn't sustain itself. A massive supply chain supports it with a population much larger than that of Shenzen behind the supply chain. Think not only of the farmers who grow the food, but also the supply chain of food production, the plants that process the fertilizer, all the components and raw materials of the factory machinery that processes grain, the transportation infrastructure that ties it all together and the supply chain of that transportation infrastructure. It just goes on and on and on.

[BurningFrog]

As people more technology makes people more productive, that number will shrink.

[jeremyjh]

This estimate was based on a projection of our technological state 10,000 years or more from now, but without super-intelligent AI or FTL travel. So human settlements are all over the galaxy, but they are fragile and disconnected from each other by hundreds and thousands of years of travel in slow time. One of the problems in this world is a civilization may have crumbled in the time it took to travel there and be in such a reduced state that its incapable of refitting your ship, leaving you marooned.

https://en.wikipedia.org/wiki/A_Deepness_in_the_Sky

[RajT88]

I can imagine now the argument, "bit who will pay for it?".

Ironically the periods of time in which we did not care about cost and really did start leaving Earth, it was driven by (drumroll) the existential threat of a particular technology.

[gremlinsinc]

Ironically, the only way we ever do that - probably, is by using Super AGI to solve many physics problems, and get us warp drives or wormholes.

[Pxtl]

AGI also provides a way to have a payload intelligence that's capable of surviving interstellar travel without warp - it can live for a thousand year trip in a shoebox.

Which is why i think our culture and stories might make it to the stars, but I'm pretty sure we won't.

[evrydayhustling]

> we need to fundamentally change how we govern humanity to mitigate existential risks

In both cases, which "we" can you possibly mean? Like, today's voters? worldwide? Today's HN users? The CCP? The idea that any "we" may or must choose how to govern humanity is a category error.

There are lots of forces other than markets to appeal to - nations, industry groups, religions. But any of them will be in a power struggle, except to the degree that they can show a legitimacy to control. Who has the most legitimacy on this subject?

[blagie]

> There are lots of forces other than markets to appeal to - nations, industry groups, religions.

Most of which suffer from an analogue of market forces. For example, elections are won by playing the "getting elected" game near-perfectly. That leads to behavior which no one might want, but people are forced into.

> The idea that any "we" may or must choose how to govern humanity is a category error.

The category error is that not making a choice isn't a choice. Right now, we're governed by complex dynamics, which are at least as restrictive as choices we might engineer.

The book "Dictator's Handbook" gives a nice overview of the dynamics which govern us. It's a game-theoretic (but non-mathematical) analysis of political systems, including corporate and democratic governance.

[keithalewis]

Paging the Unabomber. Ted to the white manifesto phone.

[dang]

Could you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. We've asked you more than once to stop, and eventually we have to ban such accounts.

If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.

[raviparikh]

Yes, Cowen argues against the idea we can anticipate consequences of technological change, and the specific consequence he focuses on is the idea of existential risk stemming from AI. He says because technology is unpredictable, we shouldn’t try to predict the type of risk imposed by AI, and we should mostly just accept that this change will happen and cope with it afterward. This stance is what I was arguing against in the post.

> But to wish to halt AI advancement requires an unhealthy mix of pessimism and overconfidence in your predictive powers.

I did not argue for this in my article!