[javajosh]

Galactic utility is ridiculous. But so is the other extreme, to act only on short-term utility. The asymmetry is that most humans don't need any encouragement to think short-term, but they do need some impetus to think long-term. But it takes wisdom to balance the two concerns. And yes it is a huge downside to any strategy that delays gratification that it can be "hacked" for evil, which is a problem more general than EA or "longetermism".

[woodruffw]

Assuming utilitarian ethics for a moment, I agree!

Maybe someone more familiar with the EA movement can answer this: why has rule utilitarianism disappeared from EA rationales for behavior? That seems to me a very natural check on these otherwise extreme (long and short-term) leaps of reasoning.

[hither_shores]

I don't know about "EA rationales", but rule utilitarianism is unpopular among philosophers largely because it's believed to collapse into act utilitarianism. The phil101 caricature is that rule utilitarianism says "no stealing, because 'no stealing' is a rule that increases utility" and act utilitarianism says "steal if you think it maximizes utility" - but on the one hand "no stealing, except for bread to feed your starving child" is an even better rule, and so on; and on the other hand not incorporating uncertainty into your decision procedure isn't being a good act utilitarian, it's just being an idiot. And so in practice they endorse the same decision procedures in pursuit of the same end.