I'm not saying you have to be Gates-wealthy to do philanthropy, I'm saying dropping your career to do nothing but philanthropy doesn't scale or sustain if you're significantly less wealthy.
Taking a high-paying job, hiring your already well-paid ex-colleagues, and working on toys for the upper-middle class wouldn't strike most people as 'giving back to the less fortunate'.