As a hiker, I would much rather step aside for an occasional biker than deal with the huge, easily startled, smelly beasts that shit all over the trails.
As a hiker I would much rather step aside for a slow moving well trained animal then dive out of the way for a mountain biker going downhill way over the posted speed limit around sharp corners.
That's just everyone's preferences rather than anything objective then. Perhaps those horseback riders damage the serenity of the mountain bike riders or the hikers by crapping all over the trail. Well, their horses, I mean.
Also, remember that horses are not an indigenous species in recorded history in North America.
People aren't indigenous to the americas either, having introduced ourselves but a dozen or two thousand years ago.
I don't think one should consider nativity very much when deciding if something is good or not in a given context. More what the impact is, irrespective of nativity.
Never the less, bikes are even less native than horses... At least horses are organic.
It's also good to remember that deer, bear, and untold numbers of other species crap all over the trail also. That hasn't ever been an impediment to my backpacking.