There was a blog posted to HN years ago describing a self hosted email setup in detail, and this was indeed the main issue. Everyone he emails is on a small number of big companies, and most of them don't like his server.
"After self-hosting my email for twenty-three years I have thrown in the towel"