[butyEah]

Hardware matters most. No matter how clever there’s no storing such large parameter sets on an Intel 286 with 4MB RAM.

No matter how clever the programmer there’s no encoding GPT4 with that. It was the hardware constraints that required programmers to be clever to begin with. These days it’s much more “copy paste the math directly because our data set is so robust and our hardware and networks so performant clever low level hacks don’t matter.”

Especially at big tech where they’ve used their own AI to guide them; the ability to just ask an ML system to simplify math has existed for a few years now, we’ve all seen how clever outputs were set aside for safe linear hacking.

Truly clever work is occurring in more traditional sciences like chemistry and biology these days.