Type inference systems generally have crumby error messages, and are harder to reason about all around. The best type inference systems are those that allow inference within a function definition, but the function signature requires explicit annotation. This keeps the inference local, which is easier for humans to read about. Good tooling can make either system easier to work with, but tooling is much less important in the type annotation case because the cost of adding annotations is trivial (contrary to your "torment" vocabulary) compared to reasoning about (non-local) type inference.
In general, this reduces to the principle that concrete is easier to reason about than abstract. The type signature of an unannotated function in a type inference system is maximally abstract, while (especially in practice) the signatures for functions that are manually annotated are more concrete if not fully concrete. There are still problems in non-type-inferred systems with programmers who try to be egregiously abstract, but these are fewer and farther between.
In general, this reduces to the principle that concrete is easier to reason about than abstract. The type signature of an unannotated function in a type inference system is maximally abstract, while (especially in practice) the signatures for functions that are manually annotated are more concrete if not fully concrete. There are still problems in non-type-inferred systems with programmers who try to be egregiously abstract, but these are fewer and farther between.