Then you set some env vars, compile a bunch of applications for hours and then realize that your env vars were wrong or you want to change them for some reason and you have to start compiling all over again.
Unless you want some console-only LFS. But that's not me.
I'm confident that that was hyperbole, but the overall idea is not wrong. It takes much less time to install a pre-compiled binary from a .deb or .rpm file than it does to configure and compile source code, even if you have an especially powerful CPU to hand. Why? Your distro has already done the hard work of configuring and compiling for you.