Properly managed capital gains are taxed at ~15% or less. One should hope that by the time you accrue $50 million your capital gains are properly managed.
How do you get to your 15%? Long term capital gains in the US are taxed at 20% + 3.8% net income tax + state tax. In a city like New York, you're talking close to 40% depending on your tax bracket.
That was just an example to make the point clear which varies from jurisdiction to jurisdiction. But even in your 15% example the long term effects are very significant.
Your point has been noticed time and time again. It's called "a nice problem to have." A man who is living paycheck to paycheck would love to be able to just dip into his investments and toss away $20K without at all affecting his lifestyle.
Money is power. I can't request a sit-down meeting with my own senator or local representative and expect my request fulfilled. Paul Graham can, and so can everyone in his wealth class. This is a problem, because Graham is not a constituent of either of them.