[cma]

Flash attention was also a set of common techniques in other areas of optimized software, yet the big guys weren't doing the optimizations when it came out and it significantly improved everything.

[whimsicalism]

yes, i agree that low-level & infra work is where a lot of deepseek's improvement came from