[bostonsre]

Do you think open source AI could also pose a risk to humanity and if so, how does it compare to the risks of malicious corporations or governments? It seems like open source AI has been accelerating rapidly and gaining tremendous steam and could potentially surpass or maybe just keep parity with corporations that constantly ingest open source innovations. Whatever open source produces could just be ingested by those bad corporations and governments. It seems like it would be pretty hard to regulate either private or open source AI at this point and it kind of seems like it could be an unstoppable runaway train. If AGI is controllable, maybe open source at the forefront would allow us to get to a mutually assured destruction like state where all governments are at parity.

[mrshadowgoose]

> Do you think open source AI could also pose a risk to humanity and if so, how does it compare to the risks of malicious corporations or governments?

Absolutely yes, and for the very same reasons. AGI would be an tremendous amplifier of human intent, even if that intent is provided by some segment of the general public. Sadly there are still very many people around today, that if given the opportunity to do so, would eagerly exterminate entire segments of the population.

Open-source succeeding in parallel is still my preferred outcome, as it would at least give the common person some chance of control as this all plays out. Admittedly, having thousands of fingers on humanity's self-destruct button isn't a great outcome either.

> It seems like it would be pretty hard to regulate either private or open source AI at this point and it kind of seems like it could be an unstoppable runaway train.

I don't think this is the case, but I hope I'm wrong. The compute required for training (as opposed to fine-tuning and inferencing) of foundational models is specialized, tremendously expensive and needs to be physically co-located with ridiculously quick interconnect. If a government chose to restrict training of new models, they absolutely could, as the physical footprint is ridiculously hard to hide, and all the purchases of training-relevant accelerators could be tracked down.

[bostonsre]

> Open-source succeeding in parallel is still my preferred outcome, as it would at least give the common person some chance of control as this all plays out. Admittedly, having thousands of fingers on humanity's self-destruct button isn't a great outcome either.

Yea, I'm leaning that way as well and have the same worry. There are a lot of knuckle heads on the internet, I'm not sure purely mutually assured destruction would stop the destruction. I'm not at all sure what that point looks like but maybe there will be some large strides in defensive capabilities so it would look more like MAD + really good anti-air defense, but then someone would probably build a nuclear torpedo or a hypersonic missile. Whatever happens, it probably won't be boring.