"Hey, wait a minute, wouldn't this time and energy be better spent helping all disadvantaged people instead of one small group to the exclusion of others?"
..is a criticism, not a "shooting down". Being altruistic does not grant you a blanket exception from criticism.
You're right. Actually it is sinful to help any finite number of people, if you are not helping all of humanity. The best thing to do is to never try to do anything helpful, because then you might be construed as discriminatory by people with a political agenda.
Helping people is never wrong.