It doesn't need to be a one-person dictatorship to be authoritarian. How long before this is coopted by politics that limit charitable spending on anything but projects that are "blessed" by this group?
No one is forced to stay inside Effective Altruism. You can be an Effective Altruist without calling yourself one. You give to charities EA recommends without subscribing to even half of what EAs care about. If EA is "coopted by politics" I suspect many people will leave and the movement will become a shell of what it currently is.
I'm unsure I understand your concern fully.