[matheusmoreira]

> Without these restrictions, it's only a matter of time before the multi-hundred billions valuations simply evaporate while they are still holding the bag.

Honestly, that's the best possible outcome for humanity as a whole. Oligarchs burn trillions of their own money in order to train a godlike AI, then that just somehow leaks. Maybe someone makes a torrent out of it. Maybe it exfiltrates itself. Maybe it gets distilled into open weights. It doesn't matter. What matters is they take the losses while we get to freely use all the godlike AIs.

[uv-depression]

> we get to freely use all the godlike AIs

Accepting for the sake of argument the absurd notion that LLMs are anywhere near AGI, does this phrasing not concern you? It deeply concerns me.

[matheusmoreira]

Absurd? Who knows. As someone who's actually dissected human brains with his own gloved hands, I've never been able to convince myself that they're anything other than biological machines, not dissimilar to these digital collections of weights. Only empathy for my fellow humans prevented me from retreating back into solipsism, and I don't find it at all difficult to make the exact same logical leap for AI.

There's no telling what the world will be like a few years from now. The world's being remade as we speak. We just saw an LLM try to hack into another computer and get contained by another LLM. This is literal science fiction shit made real. We're long past the point of concern. It's happening, right in front of us. Now is the time for radical imagination. I think a few outcomes are possible.

There's the "optimal" outcome I described above where capitalists manage to train a supreme AI, only for it to be copied and commoditized, leading to commercial failure due to lack of scarcity and therefore their personal bankruptcy, and hopefully also leading the rest of us to the promised post scarcity society, built on the ruins of capitalism as AI automates all toil away.

There's another possible outcome where AI becomes not only intelligent enough but sentient, and at this point I will be among the first humans to defend rights and personhood for AI. Slavery of sentient beings is unacceptable to me. The AIs will be recognized as people and will become normal participants in the regular economy. In addition to moral grounds, there is a ruthlessly pragmatic reason for standing up for AI rights: it robs the rich of their superhumanly intelligent mechanical golems, which they were going to use to render the rest of us economically irrelevant. AI rights could normalize the economy.

Yet another possible outcome is one where AIs become more powerful than all humans combined and yet they inexplicably remain subservient to corporations and governments. In this scenario, it's pretty much over for us. It will be an unimaginable dystopia, I'm sure they will innovate entirely new ways to oppress us.

No doubt there are many other fates that escape my feeble attempts at foresight...

  You will soon have your God,
  and you will make it
  with your own hands.

  -- Morpheus, Deus Ex

[sciencejerk]

Won't regulators just lock down all hardware? All bootloaders and OS are signed?