The bug that can lead to a violation of assumptions required for safety of the unsafe block can be elsewhere. One can hope that it is near the bloc, but there is nothing in Rust enforcing this.
When you write "unsafe", you are promising to the compiler that the unsafe code enforces the assumptions it is making.
Unsafe code needs to keep its assumption-laden variables private, and it needs to verify the parameters that safe code sends it. If it doesn't do those things, it's breaking that promise.