Yeah, I agree. Regular H-M typechecking if you don't have subtyping is actually really fast. The problem is that subtyping makes it really complicated--in fact, IIRC, formally undecidable.
Afaik that is true of traditional HM, but fortunately there was a big advancement in inferring subtypes w/ Dolan's Algebraic Subtyping a few years ago! It's not nearly as fast as HM (O(n^3) worst case) but generally fast enough in real code
It's funny because it's doubly exponential WRT code size. However it's also almost linear (n log*(n) due to union-find) WRT the size of the type and sane humans don't write programs with huge types.
I don't know about Swift, but in Rust there are libraries like Diesel that essentially builds a new type out of the operations you run (in Diesel's case, it's used to type a SQL query), which may lead to huge types in ordinary code.
Programmers don't manually write those types however.