[fxtentacle]

"The handcrafted optimizations just concern speed"

That is the goal here. Efficient execution on mobile hardware. Mobilenet v1 and v2 did similar parameter sweeps, but perform much worse. The main novel thing about v3 is precisely the handcrafted changes. I'd treat that as an indication that those handcrafted changes in v3 far exceed what could be achieved with lots of compute in v1 and v2.

Also, I don't think any amount of compute can come up with new efficient non-linearity formulas like hswish in v3.