If you have the budget, I think it's better to have different sets since the activities are so different you'll probably want drastically different things out of them. I use a pair of bluetooth earbuds when I work out because they're convenient and built to take sweat and inclement weather, but at home I use a pair of over-ear wired headphones with cloth earcups.
I do similar, use amped headphones at my desk but cheap Bluetooth headphones for working out. I actually enjoy it because I don't have to worry about being careful with the headphones and I get to listen to a more "fun" sound signature for a short while. Chinese audio has really progressed well and it's really impressive what you can get for $30
I think it's more that if I spend $200 on headphones that proclaim to be good for fitness, I expect them to sound great as well and in general be worth the $200.