[CamperBob2]

I don't know about the specifics of mikewarot's point below, but I think he's close to verbalizing a fairly-important truth: there is no reason whatsoever to think that Von Neumann machines are the best way to implement neural networks. There are lots of reasons to think they aren't, starting with the VRAM bottleneck. The impressive results that have been achieved so far have almost certainly come from using the wrong tools. That's cause for optimism IMHO.

Digital computer architecture evolved the way it did because there was no other practical way to get the job done besides enforcing a strict separation of powers between the ALU, memory, mass storage, and I/O. We are no longer held to those constraints, technically, but they still constitute a big comfort zone. Maybe someone tinkering with a bunch of FPGAs duct-taped together in their basement will be the first to break out of it in a meaningful way.