[kingstnap]

This is actually really neat, I had an agent create some experimental probes and check out whats going on here and it seems like a pretty cool project. I'm gonna need to digest how this might be useful but I think this could be cool for some sort of FPGA uses perhaps.

> this is not going to beat XLA for standard deep learning workloads. XLA has years of hyper-specific optimizations for linear algebra on GPUs and TPUs.

I'm not so convinced about this.

It's actually really easy to write two mathematically equivalent formulations in Python of something basic that have over a 2x performance difference in them after jax jit.

XLA is not that smart. And I'm not talking some niche nonsense I mean simple matrix multiplication graphs and residual connections on the CUDA backend.

[kingstnap]

Just for receipts I span up a basic test for XLA:

Can you figure out that 16 matrix vector multiplications into a concatenate is the same as concatenating first into a matrix matrix operation which can go on the GEMM. This is like the most basic thing you can imagine doing.

Turns out no it doesn't and there is a 23% performance difference by moving the concatenate up in the python code in my specific test.

Not to say that it didn't recognize it. I had an agent look at the XLA and the graph actually does a partial fusion into a sum and stack. But does not realize the whole thing is just a GEMM.

In more complicated examples the differences you can get can be much larger.