[ifdefdebug]

For me: when General Purpose Intelligence (GI) systems can build better GI systems.

Note that it's not proven that this is even possible: the only GI systems we know about (organic brains) so far have failed to do so, despite trying hard.

And I don't believe those organic brains are anywhere near to succeed, because I doubt seriously that the current approach (programs running on transistors) is capable of producing such a thing.

[willis936]

If it's down to a physical processing power limitation then I don't see any showstoppers to humans making a bigger version of a brain. It's not very elegant to just do it bigger, but it at least highlights an avenue to improving existing systems.

I mean we got here without any sort of intelligent hand, surely there is no mystical barrier to the improvements we can make.

[ifdefdebug]

Well just bigger might not cut it, the biggest brain we know about already has about almost 10 kg of weight (belongs to a whale). We want better, not bigger.

You say it yourself, there was no intelligent hand involved in the creation of brains (but another process called evolution). This just supports my point that a GI system creating a better GI system to our knowledge has not yet happened.

I agree there's no mystical barrier, but maybe a conceptual: is there a limit to our cognitive capabilities that prevents us from designing something better than us? I don't know, but it hasn't been disproved yet either.