Rogan was labelled alt-right after he interviewed Gavin McInness and Alex Jones again (a comical interview btw). That may have been cancelled out by him interviewing Bernie Sanders though.
Rogan is aesthetically a bro, that's all this comes down to.
If he were more 'soy' and less 'bro', while having the same mostly-liberal opinions, he'd get an entirely different reaction from the elite cultural left.
And I'm too American to give you a good analogy :) In England I think they'd say 'chav'?
Think culturally lowbrow vs cultural elite. Rogan came up as a cage fighting commentator. In Europe there would be a lot more techno blasted out of a much smaller car.
A stereotypical bro is someone who says "bro" a lot, like to do chest bumps with their others bros, scream about sports while shotgunning a beer, and talks with his bros about women as sex objects.