[galaxyLogic]

I think Singularity is hype. What does it mean? Machines do something we can not understand? So talking about Singularity is really talking about something we can not talk about because we don't undertand what we are talking about?

Wittgenstein said "From what we can not speak of, we must be silent about". That sounds like a tautology but I think there is a deeper meaning behind it.

It means simply that once you start talking about what we can not talk about, you are already talking about that and therefore it is NOT something you can NOT talk about. Clearly we can talk about it because we are already talking about it. And therefore it is not something that can not be talked about. That is a paradox, a bit like Godel's, but something that doesn't contradict itself.

[XorNot]

You got it in the third sentence and then dismissed it for some reason?

That's exactly what the Singularity is: it's the transition point beyond which meaningful predictions aren't possible.

In a black hole it's the center where relativity breaks down.

In AI it's the point at which non-human intelligence no longer requires human intelligence for self improvement: after which predictions of the future become somewhat meaningless.

In the human lives experience, I would argue its like having your first child: you can know what's coming, study the theory, know everything to expect and youre still you on the other side...but you can't really know what will happen till you get there.

[galaxyLogic]

Good definitions. Would you then agree that when we reach the point where AI can improve itself without our help, it is still possible to make predictions, about it?

I think we are already in the stage where AI can and does improve itself. But why should this stage be called "Singularity"? Like a Black Hole? That sounds like hype to me.

When AI can improve itself, wouldn't it still be able to explain to us how it has improved itself? If it can not it still has a lot of improvement to do.

Or are we saying that some things are "unexplainable" and AI will discover such things without being able to explain to us what they are? That sounds like mysticism, or hype to me. Or religion. We can not explain God, right?

[XorNot]

The history of predicting innovation even a decade out is full of misses, even then the actors involved were all human.

It is significant that if machine-intelligence can improve itself, then there's obviously a transition point at which the bulk of innovation shifts from being done by human minds to being done by machine minds - and if nothing else that's a "phase" transition in technological progress, namely that the future of technological development is no longer the product of human innovation but of our created machines, hopefully acting according to our desires.

The technological singularity is that: the pace of innovation separates from the ability of human minds to keep up with understanding it. We can barely predict our own innovation: we would be unable to usefully predict innovation happening by alien intelligences operating faster then we can think.

Now of course where that line exists, or if it's even plausible, are separate questions (there's a reasonable argument that physical manufacturing is likely to be a serious limiter for a good long time still at the very least).