[jrimclean]

You're neglecting the emergent effect of neural networks. Fully understanding the output of one or more individual neurons hardly tells you anything about how the network behaves has a whole. From that standpoint, neural networks are nearly impenetrable. With human written software you can at least trace through the source code and get an idea of what it's doing. Imagine being handed a large matrix of numbers, being told it represents a neural net, and then being asked to find out what it does. Good luck!

[rwallace]

Having been a programmer for thirty-some years now, I can assure you, impenetrable emergent effects in human-written software are very much a thing, and fully understanding the operation of one or more individual lines of code doesn't necessarily tell you anything much about how the program behaves as a whole. Other things equal, there is some tendency for human-written software to be easier to understand, of course, but it's very far from always true.

Think of it this way: if it were impossible to understand opaque, highly emergent systems designed by nonhuman processes, we might as well shut down every life sciences lab and toss all the biology textbooks into the recycle bin. In reality, of course, we can figure out how biological systems work, if we care to put in enough effort, and artificial neural networks are not only orders of magnitude simpler, but have the enormous advantage that we can run them on digital computers, which at least gives us full access to all the raw code and data.