[repsilat]

I'm not sure I buy this.

If

- your network structure is fixed, and

- you always evaluate the matrix in a given order, and

- you are careful/smart about how you train the weights (Ok, I haven't thought through the ramifications here...)

then I'm not sure you care much about either commutativity or associativity. Maybe a lack of associativity makes backprop impossible, and maybe commutativity makes "Google deep dream" impossible (no idea), but I don't quite agree with the "composability" objection to a lack of associativity and I don't understand the objection to a lack of commutativity sorry.

[AstralStorm]

The problem is with gradient propagation which requires approximately associative mathematics. You will be training your network something else than you wanted to.

Toy evaluate the network in two orders - forward (use) and backward (training).

[varelse]

Floating point math itself is commutative, but it's not associative as it is implemented in current processors so you've already lost associativity from the get-go.

[danharaj]

Floating point's non-associativity is certainly a problem for composability and stability of neural networks, you're right. However this non-associativity is well behaved in the sense that there's an ideal arithmetic we wish to approximate and we have techniques for mitigating the discrepancy between floating point and that arithmetic.

The non-associativity of the octonions is fundamental to their structure, not something to be worked around. In particular, there's no way to consider an octonion-valued network as comprising several layers plugged in serial.