[api]

> a much younger Eliezer Yudkowsky cautioning that Doug Lenat should have perceived a non-zero chance of hard takeoff at the moment of its birth

Why is Yudkowsky taken seriously? This stuff is comparable to the "LHC micro black holes will destroy Earth" hysteria.

There are actual concerns around AI like deep fakes, a deluge of un-filterable spam, mass manipulation via industrial scale propaganda, mass unemployment created by widespread automation leading to civil unrest, opaque AIs making judgements that can't be evaluated properly, AI as a means of mass appropriation of work and copyright violation, concentration of power in large AI companies, etc. The crackpot "hard takeoff" hysteria only distracts from reasonable discourse about these risks and how to mitigate them.

[thom]

Perhaps we can disagree on the shape of the curve, but it seems likely that ever more capable AI will enable ever more serious harms. Absolutely true that we should counter those harms in the present and not fixate on a theoretical future, but the medicine is much the same either way.

[TeMPOraL]

> Why is Yudkowsky taken seriously?

  Trivialities    Annoyances    Immediate harm     X-Risk
  |------------------------------------------------------|
         \----stuff you mention-------/
                                 \---stuff Eliezer------/
                                      wrote about

> The crackpot "hard takeoff" hysteria only distracts from reasonable discourse about these risks and how to mitigate them.

IDK, I feel endless hand-wringing about copyright and deepfakes distract from risks of actual, significant harm at scale, some of which you also mentioned.

[rvba]

> "LHC micro black holes will destroy Earth" hysteria.

I will be heavily downvoted for this, but here is how I remember it:

1) LHC was used to study blackholes and prove things like Hawking radiation

2) LHC was supposed to be safe due to Hawking radiation (that was only an unproven theory at the time)

So the unpopular question: what if Hawking radiation didnt actually exist? Wouldnt there be a risk of us dying? A small risk, but still some risk? (especially as the potential micro black hole would have the same velocity as earth, so it wouldnt fly away somewhere into space)

On a side note: how would EURISCO evaluate this topic?

Since I read about this secretive CYC (why u can email asking for it, but source not hosted anywhere?): couldnt any current statistics based AI be used to feed this CYC program / database with information? Take a dictionary and ask ChatGPT to fill it with information for each word.

[api]

The fundamental reason that hysteria was silly is that Earth is bombarded by cosmic rays that are far stronger than anything done in the LHC. The reason we built the LHC is so we can do observable repeatable experiments at high energies, not to reach energies never reached on Earth before.

The AI hysteria I'm talking about here is the "foom" hysteria, the idea that a sufficiently powerful model will start self-improving without bound and become some kind of AI super-god. That's about as wild as the LHC will make a black hole that will implode the Earth. There are fundamental reasons to believe it's impossible, such as the question of "where would the information come from to drive that runaway intelligence explosion?"

There are legitimate risks with AI, but not because AI is somehow special and magical. All technologies have risks. If you make a sharper stick, someone will stab someone with it. Someday we may make a stick so sharp it stabs the entire world (cue 50s sci-fi theremin music).

Edit: for example... I would argue that the Internet itself has X-risks. The Internet creates an environment that incentivizes an arms race for attention grabbing, and the most effective strategies usually rely on triggering negative emotions and increasing division. This could run away to the point that it drives, say, civilizational collapse or a global thermonuclear war. Does this mean it would have been right to ban the Internet or require strict licensing to place any new system online?

[at_a_remove]

You remember ... wrongly. It's just another particle accelerator, its intention was not to produce micro black holes for study.

You shouldn't use "theory" when it comes to science unless you know what that means. Gravity is a "theory." "Theory" means that it has a working model, comes with a ton of observational evidence in line with predictions, and it has yet to be replaced by anything better. Outside of math, nothing is ever proven. Any leading scientific theory is, at best, "yet to be disproven." And it stays in the lead until something better comes along: more accurate, extending over a greater domain, etc.

Hawking radiation has yet to be observed.

And if you're worried about micro black holes, well, even an iron atom has a non-zero chance of tunneling to a micro black hole state. No collider needed.

Cyc isn't secretive, it's proprietary, the way the Microsoft codebase is, the Adobe codebase is, and so on.

[adw]

> Why is Yudkowsky taken seriously?

People like religion, particularly if it doesn't affect how they live their life _today_ too much. You get all of the emotional benefits of feeling like you're doing something virtuous without the effort of actually performing good works.