Part of it (perhaps the majority) is excessive intermediate code. But at least some of it is that Rust's abstractions are more complex/involved. The "zero cost" doesn't refer to compilation time.
That's true of borrowing, but I'm not sure if it's true for other abstractions. I imagine generics (due to monomorphization) adds to the burden on the LLVM side.