Hacker News new | ask | show | jobs
by ralfd 49 days ago
If you take AI risk seriously then Open Source AI should not and must not win. Both by evil actors (biological weapons research) and the danger of unaligned AGI itself. There are some people who would never work for the military or Anduril (automatic weapon systems), but an OS AI „without asking permission“ would be the same.
1 comments
joshuaissac 49 days ago
If closed-source AGI wins, it is not going to be much different from a safety perspective anyway, because AI capability research is advancing faster than safety research.
link
ralfd 49 days ago
Closed Source AI at least can be controlled. See the directive of the US government regarding Fable (even if one disagrees about the directive there is no doubt that it is effective in shutting it off) or the safe guards by a corporate structure (even a profit driven one). It is schizophrenic to praise Anthropic for refusing the Department of War full access to their models but at the same time root for Open Source models.

Edit: relevant Scott Alexander article from today

https://www.astralcodexten.com/p/my-ai-opinions

> In terms of bioweapons, I expect that closed-source AIs will be heavily optimized against helping with these, and open-source AI will be banned after the first warning shot (or become economically prohibitive even before then).

link
zozbot234 49 days ago
Note that warning shot in that blog post means specifically a near-disaster event (perhaps one that's just barely averted) that's specifically caused by the AI. So far we've had AIUI no significant indication of open-weight AIs being problematic in that sense, whereas one can quibble that proprietary AIs have done dumb and dangerous things.

(For example, I suspect that plenty of folks would view the recently threatened mass scan of the DN42 hobby network as an instance of misaligned agentic behavior that would have wasted non-trivial resources, and I also think that most observers would pin the specific behavior of that AI on a proprietary SOTA model, not an open one. That's clearly not a disaster-level event, but it should scare you if you're concerned about alignment.)

link