| Who said anything about thinking for itself? A thing does not require intent or consciousness to be dangerous. How many chemists have blown themselves up because they didn't realize an experiment was dangerous? How many production systems have crashed because the developer didn't accurately predict what the code they wrote will do? Alkali metals and C++ code do not require ill intent, but they will still obliterate your limbs / revenue if you build and use them wrong. One of my more tangible hypotheses is a sort of runaway effect. Economic, geopolitical, and military competitive pressures will quickly push out anyone and anything that still relies on last era human-in-the-loop processes, the same way any organization that doesn't utilize artificial lighting, electricity, and instant communication will obviously be left far behind. You have to just trust that the machine running stock market transactions will do its math right. But unlike transaction software failure modes, which quickly result in outright crashes or verifiably incorrect errors, failure modes of non-bayesian decision making software probably looks something like what happens when existing economic, geopolitical, and military decision-makers make decisions that are harmful, unethical, or otherwise undesirable for humanity. This time augmented with, if not superhuman intelligence, at least superhuman speed and superhuman knowledge breadth. |