[fooker]

Backpropagation sort of handles this. The real reason we don't have what you describe is learning algorithms have to be efficient. So, our training/inference algorithms and 'neural' network design is all targeted to be easily representable as matrix multiplications.

Maybe with some hypothetical analog computer we could have fast training of arbitrary networks.