[vlthr]

I think the author is right to argue against the claim that “the singularity must happen eventually”, but I’m not sure how applicable your observations are to the question of whether an AI singularity could happen, or whether we should be planning for that contingency.

Today, progress in machine learning is almost entirely empirical, with limited theoretical advances following later to make sense of the empirical findings. Our theoretical underpinnings are so weak we don’t even have a rough estimate of how much harder it is to make a self-improving AGI than e.g. GPT-3. Maybe it’s many orders of magnitude harder, and not even remotely solvable by taking iterative steps from where we are now. Maybe it’s just one unifying theoretical advance away.

As for waiting and hoping it happens on its own, that’s not a great description of what’s happening. A huge number of people (possibly too many, leading to short-term incentives) are trying to make improvements in any way they can think of. Progress is happening at a staggering pace —- we just don’t have any idea how much progress is needed to reach the goal of AGI.

[jqpabc123]

In the real world, most advancements are engineered. Happy accidents have occurred but nothing on the order of AGI.

There are a huge number of people involved but as you point out, it is mostly empirical. In other words, there is no real plan --- we're just hoping for a happy accident.