[frigaardj]

It seems like the author has realised that there are a few common functional programming patterns (folds, maps) that are also common ways of combining information and operating on data structures, and seen the parallel to some operations that we frequently want to do within neural networks. A 'function' is simply a thing that takes another thing and produces a third thing. This doesn't seem that revolutionary or insightful - do these ideas give us any extra knowledge about neural networks or is this just a nice parallel?

[jeremysalwen]

I think that the theory is pretty much already clear to people who have a lot of experience with neural networks. I think the contribution is to explicitly write it down in a clear and concise way for people who it wouldn't have otherwise occurred to.

[frigaardj]

Fair enough.

[noelwelsh]

If this formalism works out then a few interesting directions spring to mind:

- we suddenly have a huge new toolbox from FP / category theory that we can apply to understanding and extending DNNs. E.g. what happens if we apply differentiation in the manner of zippers to these structures? I have no idea but it might lead somewhere.

- The deep learning world gets a precise way to describe network architecture which makes communication much easier and research much more reproducible.

- With a formal model you can automate building and optimising implementations.

[ajtulloch]

https://github.com/ajtulloch/dnngraph is an application of these ideas - it's a Haskell DSL for building DNNs for later execution by Caffe/Torch.