Correct, you cannot remove a version or the whole crate unless very specific criteria are fulfilled. You can "yank" versions. That prevents people from adding the version as a new dependency, but if you relied on it before it got yanked, your build will succeed.
I wouldn't delete old versions even if I could though. My goal is to publish a rock solid library that everyone can depend on and build awesome projects with