[rwallace]

Having been a programmer for thirty-some years now, I can assure you, impenetrable emergent effects in human-written software are very much a thing, and fully understanding the operation of one or more individual lines of code doesn't necessarily tell you anything much about how the program behaves as a whole. Other things equal, there is some tendency for human-written software to be easier to understand, of course, but it's very far from always true.

Think of it this way: if it were impossible to understand opaque, highly emergent systems designed by nonhuman processes, we might as well shut down every life sciences lab and toss all the biology textbooks into the recycle bin. In reality, of course, we can figure out how biological systems work, if we care to put in enough effort, and artificial neural networks are not only orders of magnitude simpler, but have the enormous advantage that we can run them on digital computers, which at least gives us full access to all the raw code and data.