They also use their wealth to rig the system (political manipulation) to the detriment of the societies that allowed them to accumulate wealth in the first place. Don't pretend they're blameless; they're parasites killing the host.
What I'm not for is letting them rig the rules of the game through unlimited political spending and not having functional social safety nets because we don't want to offend the all the future billionaires-to-be.
Basically, money makes money and at some level you get into infinite money glitch dynasties. I am no longer convinced that allowing that to happen is a good idea.
How are they killing it? Mankind is richer than they've ever been in history, and working less hours per day. Things are great. Things are way better than 40 years ago, 400 years ago.