[xlnt]

I sent him this objection:

We already build very powerful intelligences all the time, in the form of children. But despite having a lot of raw computing power, they aren't very useful without knowledge. They need to learn a lot of things before they can live effectively, solve important problems, etc...

Artificial intelligence and intelligence are not fundamentally different things. If you want AI's that turn out better than children do, you'll need improved parenting and educational techniques, or an alternative. I haven't seen any commentary on how singularity people plan to deal with this. And if they could, why not just use that technique on human children and vastly improve the world, now?

The main difference between AI and children I've seen proposed is that the AIs can run on superior hardware. But it's not like people currently use all the hardware resources their brain makes available; maybe that will be a bottleneck in the future, but it isn't one we've hit yet.

[byrneseyeview]

It is very hard to flawlessly copy a memory or concept into a child's brain. This might be easier if we wrote the software and designed the hardware.

[xlnt]

You think the input format will be easier for AI? I don't know. We have evolved knowledge of the input format for children. Meanwhile the majority of our knowledge is inexplicit, and we are bad at turning that into vague English, let alone into something as precise as code.

Back to children, we're so good at teaching them that we manage to teach them many inexplicit ideas without ever having much understanding of those ideas.