But you can build something smarter than yourself (or at least that's what we think we can do)... So this new entity, smarter than us, might be even better at building something smarter than itself...
Good point, I don't think I've seen that suggested before. However I think motivation is a big issue. Our instincts and emotions are what drive us far more than rationality. If we decide to build an AI smarter than us and program it to really, really want to design an even smarter AI then it might essentially have no choice.
> Good point, I don't think I've seen that suggested before.
A fictional example, but Alastair Reynolds' "Inhibitors" are a type/race of machines which, while intelligent, were specifically designed to limit their own degree of sentience.