Doesn't that get into the domain of a distro like Gentoo though? (Or sort of Nix as well) - rebuild everything to precisely target your actual architecture.
Yes, and the maximally performant configurations are at odds with a usefully hardened security posture. This only gets more true as time goes on and additional mitigations pile up.