I am as close to a fan of a clear backlog as it can be, but I would also say that an insufficiently deep backlog is probably a sign that you're spending too much on refactors and cleanups.
That would be true insofar as another premise is true: large backlog means we are drowned in business work and the internal cleanups that are very important constantly get kicked down the road. I started pushing back and the backlog at least stopped growing; even managed to remove 2-3 items this week, first time in months.