[LeifCarrotson]

"There's no stopping it at this point" - Sure there is, if a handful of enormous datacenters pull the very large plugs (or if their shaky finances collapse), the dubiously intelligent machines will be turned off. They're not ultraintelligent yet.

Stopping it merely requires convincing a relatively small number of people to act morally rather than greedily. Maybe you think that's impossible because those particular people are sociopathic narcissists who control all the major platforms where a movement like this would typically be organized and where most people form their opinions, but we're not yet fighting the Matrix or the Terminator or grey goo, we're fighting a handful of billionaires.

[observationist]

I'm not saying it's technically impossible, I'm saying that in the real world, it's not going to stop. Nobody is going to stop it. A significant number of people don't want it to stop. A minority of people are in the "stop AI" camp, and the ones with the money and power are on the other side.

It's an arms race replete with tribalism and the quest for power and taps into everything primal at the root of human behavior. There's no stopping it, and thinking that outcome can happen is foolish; you shouldn't base any plans or hopes for the future on the condition that the whole world decides AGI isn't going to happen and chooses another course. Humans don't operate that way, that would create an instant winner-takes-all arms race, whereas at least with the current scenario, you end up with a multipolar rough level of equivalence year over year.

[hollerith]

The whole world decided in the 1970s not to pursue the technology of germ-line genetic engineering of humans, and that decision has stood.

People similar to you were saying in the 1950s and later that it was inevitable that nuclear weapons would be used in anger in massive attacks.

Although the people in charge are tentatively for AI "progress", if that ever changes, they can and will put a stop to large AI training runs and make it illegal for anyone they don't trust to teach, learn or publish about fundamental algorithmic "improvements" to AI. Individuals and groups pursuing "improvements" will not be able to accept grant money or investment money or generate revenue from AI-based services.

That won't stop all research on such improvements (because some AI researchers are very committed), but it will slow it down to a rate much much slower than the current rate (because the current fast rate depends of rapid communication between researchers who don't each other well, and if communicating about the research were to become illegal, then a researcher can communicate only with those researchers he knows won't rat him out) essentially stopping AI "progress" unless (unluckily for the human species) at the time of the ban, the committed researchers were only one small step away from some massive algorithmic improvement that can be operationalized using the compute resources at their disposal (i.e., much less than the resources they have now).

Will the power elite's attitude towards AI change? I don't know, but if they ever come to have an accurate understanding of the situation, they will recognize that AI "progress" is a potent danger to them personally, and they will shut it down.

It's not a situation like the industrial revolution in England in which texile workers were massively adversely affected (or believed they were) but the people running England were mostly insulated from any adverse effects. In the current situation, the power elite is definitely not insulated from severe adverse consequences if an AI lab creates an AI that is much more competent that the most competent human institutions (e.g., the FBI) and the lab fails to keep the AI under control. And it will fail if it were to use anything like the methods and bodies of knowledge AI labs have been using up to now. And there are very bright people with funding doing their best to explain that to the elite.

Those of you who want AI "progress" to continue until the world is completely transformed need to hope that the power elite are collectively too stupid to recognize a potent short-term threat to their own survival (or the transformation can be completed before the power elite wake up and react). And in my estimation, that is not inevitable.

[LeifCarrotson]

I actually got to talk to an individual at a coffee shop just last night about this issue, he's a representative of our local PauseAI chapter:

https://pauseai.info/proposal

They think that a similar solution to the genetic modification treaties and nuclear proliferation treaties can be implemented for AI training.

[hollerith]

That plus something similar to the prohibition on publishing knowledge about the design of nuclear weapons, but for machine-learning algorithms. (Inb4 "but that's ridiculous, you can't ban linear algebra". Watch us!)

[goodmythical]

right, because turning off any number of data centers is going to do anything at all but create massive pressure on researching the efficiency and effectiveness of the models.

There are already designs that do not require massive data centers (or even a particularly good smart phone) to outperform average humans in average tasks.

All you'd accomplish by hobbling the data centers is slow the growth of sloppy models that do vastly more compute than is actually required and encourage the growth of models that travel rather directly from problem to solution.

And, now that I'm typing about it, consider this: The largest computational projects ever in the history of the world did not occur in 1/2/5/10 data centers. Modern projects occur across a vast and growing number of smaller data centers. Shit, a large portion of Netflix and Youtube edge clusters are just a rack or a few racks installed in a pre-existing infrastructure.

I know that the current design of AI focusses on raw time to token and time to response, but consider an AGI that doesn't need to think quickly because it's everywhere all at once. Scrappy botnets often clobber large sophisticated networks. WHy couldn't that be true of a distributed AI especially now that we know that larger models can train cheaper models? A single central model on a few racks could discover truths and roll out intelligence updates to it's end nodes that do the raw processing. This is actually even more realistic for a dystopia. Even the single evil AI in the one data center is going to develop viral infection to control resources that it would not typically have access to and thereby increase it's power beyond it's own existing original physical infrastructure.

quick edit to add: At it's peak Folding@Home was utilizing 2.4 EXAflops worth of silicon. At that moment that one single distributed computational project had more compute than easily the top 100 data centers at the time. Let that sink in: The first exa-scale compute was achieved with smartphones, PS3s, and clunky old HP laptops; not a "hyperscaler"

[ben_w]

> quick edit to add: At it's peak Folding@Home was utilizing 2.4 EXAflops worth of silicon. At that moment that one single distributed computational project had more compute than easily the top 100 data centers at the time. Let that sink in: The first exa-scale compute was achieved with smartphones, PS3s, and clunky old HP laptops; not a "hyperscaler"

A DGX B200 has a power draw of 14.3 kW and will do 72-144 petaFLOP of AI workload depending on how many bits of accuracy is asked for; this is 5-10 petaFLOP/kW: https://www.nvidia.com/en-us/data-center/dgx-b200/

Data centres are now getting measured in gigawatts. Some of that's cooling and so on. I don't know the exact percent, so let's say 50% of that is compute. It doesn't matter much.

That means 1GW of DC -> 500 MW of compute -> 5e5 kW -> 5e5 * [5-10] PFLOP/s -> 2500 - 5000 exaFLOP/s.

I'm not sure how many B200s have been sold to date?

[trvz]

Open models barely any worse than SOTA exist, and so does consumer-ish hardware able to run them. The genie’s out, the bottle broken.

[slibhb]

Do you really think AI companies/researchers are motivated by greed? It doesn't seem that way to me at all.

Stopping AI would be immoral; it has the potential to supercharge technology and productivity, which would massively benefit humanity. Yes there are risks, which have to be managed.

[jobs_throwaway]

AI researchers are not a monolith. I definitely think that many of them are motivated by greed. Many are also true believers that AI will improve the human condition.

I fall in the latter camp, but I think its a bit naive to claim that there is not a sizable contingent who are in AI solely to become rich and powerful.

[ben_w]

> has the potential to supercharge technology and productivity, which would massively benefit humanity

The opportunities you chose to list are the greedy ones.

> Yes there are risks, which have to be managed.

How?

As a reminder, we've known about the effect of burning coal on the climate for well over a century, we knew that said climate change would be socially and economically disasterous for half a century, yet the only real progress we're making is because green became cheaper in the short term not just the long term and the man in charge of the USA is still calling climate change and green energy a hoax.

Right now, keeping LLMs aligned with us is easy mode: they're relatively stupid, we can inspect the activations while they run, we can read the transcripts of their "thoughts" when they use that mode… and yet Grok called itself Mecha Hitler, which the US government followed up by getting it integrated into their systems, helping the Pentagon with [classified] and the department of health to advise the general public which vegetables are best inserted rectally.

We are idiots speed-running into something shiny that we don't understand. If we are very very lucky, the shiny thing will not be the headlamp of a fast approaching train.

[slibhb]

> The opportunities you chose to list are the greedy ones.

Technology covers healthcare. I don't see how it's "greedy" to want to cure cancer. But on some level I guess "wanting life to be better" is greedy.

Your attitude is very European, and it's basically why your continent is being left behind. I'm not totally against Europe becoming the world's retirement home, as long as there are places in the world where people are allowed to innovate.

[ben_w]

> Technology covers healthcare.

If you'd chosen to list that in the first place, I wouldn't have said what I did; "supercharge technology and productivity" is looking at everything through the lens of money and profit, not the lens of improving the human condition.

> Your attitude is very European, and it's basically why your continent is being left behind

And yours is very American. You talk about managing the risks, but the moment you see anyone doing so, you're against it.

And of course, Europe does have AI, both because keeping up is so much easier and cheaper than being bleeding edge on everything all the time, and of course, how DeepMind may be owned by Google but is a British thing.

Plus: https://mistral.ai

Also, to be blunt, China's almost certain to win any economic or literal arms race you think you're part of; they make too much critical hardware now.

> as long as there are places in the world where people are allowed to innovate.

I would like there to be a world.

When people worry about the end of the world, they usually don't mean to imply its physical disassembly. Sometimes people even respond as if speakers did mean that, saying things like "nukes or climate change wouldn't actually destroy the planet, it will still be here, spinning", as if this was the point.

AI is one of the few things that could, actually, literally, end up with the planet being physically disassembled. "All it needs" is solving the extremely hard challenges of a von Neumann replicator, and, well, solving hard problems is kinda the point of making AI in the first place.

[slibhb]

> If you'd chosen to list that in the first place, I wouldn't have said what I did; "supercharge technology and productivity" is looking at everything through the lens of money and profit, not the lens of improving the human condition.

Bullshit. "Technology and productivity" are not the same thing as "money and profit". You're projecting your garden-variety European degrowth ideology onto what I wrote.

> Also, to be blunt, China's almost certain to win any economic or literal arms race you think you're part of; they make too much critical hardware now.

Europeans are so hilariously polarized against the US that they would prefer China, a literal authoritarian dictatorship, to "win any global economic arms race". I guess it's because China is too culturally distant for them to feel insecure over.

> AI is one of the few things that could, actually, literally, end up with the planet being physically disassembled. "All it needs" is solving the extremely hard challenges of a von Neumann replicator, and, well, solving hard problems is kinda the point of making AI in the first place.

It's not worth wringing our hands over science fiction scenarios.

[ben_w]

> You're projecting your garden-variety European degrowth ideology onto what I wrote.

Don't believe all the memes you read on the internet.

Europe isn't degrowth, "degrowth" is a mix of a meme and environmental scientists; Europe is in fact still growing: thanks to US shenanigans, even with tech stuff that we'd prefer to outsource due to the well known economic point of "comparative advantage"; and also, thanks to Russia's invasion, we sped up energy transition and defence sectors.

> Europeans are so hilariously polarized against the US that they would prefer China, a literal authoritarian dictatorship, to "win any global economic arms race". I guess it's because China is too culturally distant for them to feel insecure over.

Prefer? No. Simply look at the back of most electronics, "Designed by … in California, assembled [by Foxconn] in China" at best, at worst the entire business is unpronounceable in English. Even when you may think you've got yourself an American factory, so many of the bits arr usually made in China, or in Taiwan which is unfortunately very insecure right now. Even when you think you've bought them from a non-Chinese company, with the goal of no Chinese parts, you can find Chinese text on the production label and that you've just paid for re-badged Chinese stuff. You may have a stated goal of on-shoring, but even with the most competent leadership this would be a very hard multi-decade project. (Similar logic applies to us shifting away from your tech, but is slightly easier for us due to open source, hardware replacement cycles, and how little of "your" hardware you actually manufacture in the first place).

That doesn't make China good in any objective moral sense, it's not like China's above doing to us what was done to them in their "century of humiliation". Just, powerful.

Their power is aside from any question of should we prefer the authoritarian in charge of a democracy who threatened to invade, or the authoritarian in charge of a one-party state that's doing some genocide who wants to sell us stuff, because two things can both be bad.

> It's not worth wringing our hands over science fiction scenarios.

AI is already a sci-fi technology relative to what I had as a kid. Or indeed relative to just after the first ChatGPT was released, given what people were saying back then that LLMs would "never" do.

The idea you could talk to your computer and it would write a computer program for you that could solve a problem that you had? Sci-fi.

The idea of computer could generate, not simply find but generate, an image according to some prompt of yours? Compose a song? Win awards for its out when people didn't realise computers doing it was an option? Sci-fi so hard it's become a meme of a robot saying "can you?", as disbelief of that was expressed as a line from the film "I, Robot", 2004.

People are still arguing if these things have or have not passed the Turing test, someone has even made a game about this for Hacker News comments, I game in which I score 0, or even scored negative given I only identified false positives. Sci-fi.

And it's not just LLMs, Even just solving chess was sci-fi when I was a kid. Then it was Go. Now protein folding is solved, and thousands of novel toxins have been found by AI. And yet, when I have told AI-Laissez-faire-accelerationists stuff like this latter example, they still doubt AI is capable of doing anything dangerous.

But the worst part of it? The AI which called itself Mecha Hitler, that AI is in use by the Pentagon, the DoD is trying to bully a different AI company that doesn't want to be used for military stuff.

We're in a sci-fi future.

And remember too that making a "robot army" that can replace all human labour is a stated goal of one of the people running an AI company. Don't get me wrong, I hope he's talking out of his rear on this, but failing to plan is planning to fail.

And you've not, at any point in any of your replies, answered my earlier question: how to manage any risks from AI.

[rune-dev]

> Do you really think AI companies/researchers are motivated by greed?

Researchers, maybe not. Companies, absolutely yes.

I don’t see how you could assume the likes of Google, Microsoft, OpenAI, and even Anthropic with all their virtue signaling (for lack of a better term) are motivated by anything other than greed.