[TheOtherHobbes]

We don't really know how AI works either. NNs (for example) do stuff, and sometimes it's hard to see why.

>Taking a position that neural networks cannot ever result in strong AI is as blind as taking a position that they must.

Not really. Right now it's taking the position that there is no practical path that anyone can imagine from a go-bot, which is working in a very restricted problem space, to a magical self-improving AI-squared god-bot, which would be working in a problem space with a completely unknown shape, boundaries, and inner properties.

Meta-AI isn't even a thing yet. There are some obvious things that could be tried - like trying to evolve a god-bot out of a gigantic pre-Cambrian soup of micro-bots where each bot is a variation on one of the many possible AI implementations - but at the moment basic AI is too resource intensive to make those kinds of experiments a possibility.

And there's no guarantee anything we can think of today will work.