Some restrictions aids performance, but checking that pointers have not run wild like elphants in a porcelain shop requires extra checks, which takes time.
Only if the restrictions are static and don't need to be checked dynamically, naturally. For dynamic arrays or dynamic indices it's not always possible to do that in Rust.