[bux93]

Because both of them are optimized for hardware. Neural networks, despite the name, have very little similarity to biologics.

There's a lot of multiplication of numbers in parallel, so it makes sense to try to fit that to matrices.

Cryptography is built bottom-up, but likewise it makes sense to exploit data structures that already exist in silicon.

[PxldLtd]

While modern LLMs are a far cry from biological synapses, I do find it fascinating that if you take the highly reciprocal data of a biological connectome and unroll it into a DAG, you suddenly see motifs popping up that look similar to what we find in AI. I found this both looking at temporal unrolling of RNNs or mapping layer activation weights of a Transformer. Totally agree though, the current LLM architecture itself is driven by the need to shove all of this nicely into parallelized compute hardware.

[krisoft]

> if you take the highly reciprocal data of a biological connectome and unroll it into a DAG, you suddenly see motifs popping up that look similar to what we find in AI

That sounds interesting. Where have you heard about that? Or is this your own research?

[amoss]

In addition both have a property similar to dispersion. In crypto each change to an input bit should cascade through as many output bits as possible. In ML each output bit should depend on as much of the input bits (and hidden layers) as possible. So they both feature a similar maximization of entropy.

[winfieldchen]

> dispersion [...] maximization of entropy

This is exactly the point. I was disappointed that I had to scroll so far down the page until I saw the word "entropy." There is a deep connection between machine learning and encryption and compression in information theory. As Shannon demonstrated, the one-time pad's encrypted output is maximum entropy, and so would data compressed to the Shannon limit. Such an optimal compressor learns the underlying probability distribution of the data to represent it with the fewest bits possible, which is exactly the goal of machine learning. A trained ML model can be seen as a lossy compression of the training data. Autoencoding models make the link between ML and compression (and thus encryption) explicit.

[gblargg]

More simply, most interesting operations will involve "mixing up" of data. There's only so much you can do by applying a bunch of operations in series with a single input value.