It doesn't matter. Pick a number. Maybe you have a 650 MiB cap because you're releasing software on CD. Maybe you have a 25 GB cap because you're releasing on BD. Or 100 MB for some app store. Yet, semver only considers API compatibility, not other factors such as binary size. Any minor change could push the size beyond an acceptable limit. The point still stands.
But at a higher, more reasonable limit a tiny change is much less likely to push you over the line.
Also remember it's not the size of the dependency that matters it's the size of the portions of the dependency used less any cross-crate optimizations. So you're always going to have to be careful and test regularly.