[Iv]

I don't come from a protestant country, "the end is nigh" does not really ring a bell in catholic countries.

The technological singularity, originally, comes from a very practical observation by Vernor Vinge, who was explaining that as a SF author, once humanity reaches the point where it can make intelligent machines that improve themselves, he hits a "horizon event" that he is not intellectually equipped to look beyond.

I have a hard time finding a good counterargument to it: the emergence of self-improving AIs of human or super-human capabilities will be an extremely strong and fast change in human progress. That much is true.

So yes, the hopes one can place in it look similar to the hopes of the religious eschatologists, but using this similarity to reject the argument is just an association fallacy.

[mvindahl]

Going a bit off topic here but your remarks on religion makes me curious. I'm from a firmly -- albeit lazily -- protestant country and don't describe myself as religious in any meaningful sense, but still an interesting subject.

My reference to religious "singularities" was to the apocalypse, or however you want to name it. The End of Days. The Second Coming. Stuff like that. I thought these ideas played a similar role in all branches of Christianity, although ignored by most people for most of the time. Is Catholicism and Protestantism so different in that regard?

More on topic, I see the evolution of (better) AI as a logical extension to the computer revolution, and I see the computer revolution as equivalent in scope to the industrial revolution. So I'm not saying it's a small deal. I just don't believe a whole lot in self-improving AI as a concept all that different from what has been going on in, say, processor design, for decades, and I certainly don't believe in the idea of a runaway, self-improving AI.

To become rapidly and exponentially more intelligent, it would have to either work smarter or work harder (to use an overused phrase). Working smarter would imply that if you arranged your CPU instructions in increasingly clever ways, you could extract exponentially rising performance from your programs. That's not how I remember the assembler programming that I did as a kid; you could be clever but there were some pretty hard limits. The strategy of working harder, on the other hand, would imply that someone would be willing to supply exponentially rising amounts of energy. Again, even if some Zuckerberg or some shady government agency would be willing to play this game, you'd quickly run into hard limits.

So yeah, AI is not the first game changing phenomenon in the history of humankind but it will change stuff. Like did bronze, railroads, or nukes.

[Iv]

About the religious thing, I think the catholic mindset is that the end of the world can't be predicted and anticipated. That you should worry less about it than the afterlife. Most protestant actually think the same way but protestantism is more diverse. If 9 out of 10 groups do not care about the apocalypse, you will not hear from them. You will hear from the JW-style preacher saying the end of the world comes next week.

Also, the notion of the rapture is pretty much a protestant interpretation. Most catholics see nothing to rejoice about in the apocalypse, even for good believers.

Back on the AI. Take this scenario: we make a software that's can be as smart as a human engineer, runs in 100% realtime speed on the fastest CPU we can produce. We know it is physically possible because our brain does that.

Buy all the CPU you can buy, compared to human researchers it is cheap. Make teams of hundreds of units (or whatever number is ideal). Put some team on the task of improving the parallelization of the AI program, some on the task of designing a faster CPU, some on the single-core optimization of the code, some on the theoretical research to improve the underlying algorithms. You'll quickly gain exponential results. Actually you will get a fast exponential rise of global intelligence.

The number of researchers will not show an exponential function but a steep step: almost overnight the number of minds working in R&D will change its order of magnitude. This is disruptive. All the currently researched and underfunded problems will be solved orders of magnitude faster. The small problems not worth solving nowadays will see teams of hundreds of top engineers devoted to them.

> The strategy of working harder, on the other hand, would imply that someone would be willing to supply exponentially rising amounts of energy. Again, even if some Zuckerberg or some shady government agency would be willing to play this game, you'd quickly run into hard limits.

Science fiction authors searched for that hard limit but do not see a roadblock until all of earth's matter has been converted to computronium and in a matrioshka brain, making us reach type II on the Kardashev scale. And I agree with them.

Yes, more energy will be required. So what? With enough research to improve generation and automation of construction and deployment of power plants, we can have an exponential rise in energy for quite a while. And if we mastered fusion, we could even go beyond type II civilizations.

> So yeah, AI is not the first game changing phenomenon in the history of humankind but it will change stuff. Like did bronze, railroads, or nukes.

I don't think humanity ever had an event of the same potential. Maybe the birth of civilization, when they started gathering in cities, but even these changes were pretty slow, and their effects happened at the time scale of human reproduction, that has not changed since millions of years. The changes we are talking about rely on our speed of CPU power production, which we improve constantly and that this process will accelerate.