| > could be catastrophic for humanity, I do not. How does super human intelligence, on it's own, represent any sort of risk for "humanity?" > then the recipe for how to make AGI is of itself an infohazard [...] nuclear weapons I know how to make nuclear weapons; however, I cannot enrich the fuel enough to actually produce a working version. > and yet we regulate its access Does that actually achieve what it claims to achieve? > Do I trust OpenAI to make all the morally right choices here? OpenAI has a language model. They do not have AGI or anything approaching AGI. > No, but I think their incentives are in fact more aligned with the public good We could debate that, but for your assertions to hold any water, we'd have to agree that they're incapable of making mistakes as well. Far easier to skip the lofty debate and recognize the reality of the world we live in. |
It does not; I was speaking to AGI. But assuming you were referring to AGI as well, you don't have to think very creatively to consider scenarios where it would be harmful.
If you have an agent that can counter your every move with its own superior move, and that agent wants something – anything – differently from what you want, then who wins? Maybe it wants the money in your bank account, maybe it wants to use the atoms in your body to make more graphics cards to reproduce itself.
Think about playing a game of chess against the strongest AI opponent. No matter which move you are considering playing, your opponent has already planned 10 steps ahead and will make a better move than you. Now extrapolate outside the chess board and into the realm where you can use the internet to buy/trade stocks, attack national infrastructure, send custom orders to chemists to follow whatever directions you want, etc.