Without the type annotations, you're expected to handle dynamic type checks yourself. With type annotations, you might think that tooling does it for you, so you might omit the dynamic checks. But the tooling doesn't necessarily check all your callers, and you end up with unexpected input.
This is partly what the Typed Racket writers mean when they say that most gradual typing systems like this one are not sound.