It's a big deal when maintaining a fork. It's tempting to merge upstream commits back into your branch, but you should always rebase and keep a clean patch set in such situation.
I don't understand how you are using rebase with a fork. Are you rebasing all of the changes in your fork on top of upstream? That list of changes will grow larger over time; after a year, you'd be rebasing hundreds of commits whenever you want to merge upstream.
A friendly fork generally has a natural limit on how many commits there will be on top. Personally I've done this with ~200 feature commits and it's not a big problem (as long as you use incremental rebase of course).
Of course if you're planning a hard fork, merges may be unavoidable. But I've seen too many Franken-linux-kernels which were forked from 4.x with periodic merges whose correctness is impossible to verify. Inconsistencies eventually build up with each merge.