[glenneroo]

Did you read the article or letter? They list a lot of very valid reasons. I won't bother quoting because I would just be copy/pasting the article (which is very short).

For a more detailed list of reasons, go read what AI alignment scientists think, since they have been working on how to align an AI in order to not "turn everything into paperclips" since the 1970s and it seems as though many are rather skeptical about us having any future if we keep up at this rate (spoiler: many believe the end of humanity will occur soon after the creation of any superintelligence): https://www.alignmentforum.org

My personal takeaway: continued development might mean the end of humans and maybe even our planet (if an ASI can deploy nanotech to convert everything into substrate). As it already stands, nobody knows why LLMs work as well as they do - they are already a black box. Sure, plenty of people can explain the math behind each of steps involved: training, matrices, transformers, inference, et al. but it's still a big black box which spits out "magic" answers. You can't just drop a breakpoint inside a model during inference to see what's going on, you will just get a long list of unintelligible floating point numbers at any step of the process.

Your question about regulation is valid... but something needs to be done. I feel like we're standing very close to one of the Big Filters.

Comparing AI/AGI/ASI to anything we have currently seen is probably pointless, they are worlds apart. Would you bother comparing a smartphone to a book? The rate of AI/ML progress is sufficient to measure in hours instead of months or years.

[kromem]

It could also present the only realistic way we avoid the many other human-ending threats we've introduced ourselves.

This isn't a one sided coin flip.

You have nuclear war, emerging bioweapon capabilities, global warming, pollution, and a number of other existential concerns. And yes, AI is one on that list.

But how many solutions to those other items do we realistically have right now? Because it keeps seeming to be that nothing is getting done outside people pointing out the sky is falling.

Quite possibly the ONLY real, well funded, viable solution to these issues would be a straight up deus ex machina being pulled out of our rear ends.

So yeah, it's a scary coin flip that could add to the massive list of other things inevitably going to kill us all, or maybe end up being the coin flip to negate the rest of that list.

So a letter that only points out a possible threat model isn't genuinely pursuing discussion over the ethical considerations - it's just fear mongering. And looking at signatures much further down the list of "concerned parent" or "fearful citizen" it's having the intended (and very sketch) effect.