Sounds like a great motivation for academic researchers to find a way to train LLMs with less compute. Or maybe invent something better than transformers. A brain trains on 20 Watts after all.
That's a hardware difference. Brains run at a very low clock speed and make up for it with massive parallelism. They also don't suffer from the vonn neumann bottleneck - today's computers spend most of their time and energy shuffling the network in and out of memory.
I believe that better hardware architectures will have more impact on AI than better neural network architectures.
Today's computers don't spend most of their time in I/O. Average CPU runs idle most of the time. I/O does not require CPU power. DMA exists since the 80s or whenever. Software and segmentation limits such as operating systems and traversal between execution rings are not the chemical hard barrier brains have.
You're correct that our architecture isn't adequate and biggest achievements lie there. I/O is not the problem, in fact, we have faster I/O. Because our I/O is dumb. We can place massive amounts of data in a linear memory buffer. But brains use massively associated memory structures. I/O of a network packet is easy. Associating that packet with preexisting context (such as TCP connection) is not that easy, requires structures, algorithms, memory locality, threading correctness, and procedural computing steps, because we abstract the context over a series of flat data.
If you're working on a subject hard, just a random flying info about something else that concerns you might trigger "I can't think about that right now" reaction in your brain, but the information has been digested. The packet has reached the adequate layer 7 ingress buffer just like that, but you don't want to context switch to the respective application intentionally.
There is also an elephant in the room and that is the native language, which shapes the way we think and process information. Imagine a CPU receiving an automatic microcode update the same moment when you as a programmer defined an abstract TCP stack in C or assembler, so it can optimize itself to the point of being able to switching to "thinking in TCP" mode.
Pretend you have any hardware you want, today. What would you do with it? What model would you train? How do you know available hardware is the bottleneck and not model architecture?
Neuromorphic Computing is already a thing, and Intel's already developing chips (Loihi 2).
But it's not as powerful as GPU's yet, and it only runs spiking neural networks.
I doubt it. FPGAs are super inefficient in transistor count in exchange for being dynamically programmable. I suspect a better architecture will be taped out like any other chip.
I believe that better hardware architectures will have more impact on AI than better neural network architectures.