Some people block ads because they don't like the tracking that comes with it.
When asked to add a site to the whitelist to support it the downside is that you will be tracked.
I don't know about everybody else but script bloat, and its implications for performance in generally and scrolling smoothness in particular, are the main reason that I block ads - the visual distraction is a secondary factor.