The problem is the preceding if clause. You say that you do this if you find it uninteresting, or if you have no expertise in the field of the paper. Both of these are qualifiers of you, not of the paper, so indeed it seems harsh to punish the authors for the bad referee selection of the editor.
He'll decline the request or say it should not be accepted. Declining the request is just fine, and a reasonable thing to do. There are many mathematical results that are in fact uninteresting (to just about anyone) and it's fine to suggest those be rejected from a good journal.
I've had many paper rejections, none because of false or uninteresting claims but almost all because "it's not good enough for this prestige journal" or "it's too specialized for this journal". I expect a certain number of those, because I submit to reach journals now and then and hope I get a reviewer and editor that like the work :) If I wanted certain publication immediately I'd submit to lower-tier journals first, which I have also done when I just needed something out.