I think the more normal thing to do would be not to buy a TV and just don't make a meal out of explaining why. 'I would never use it, ' is pretty sufficient
I think you're living in a bubble in this regard. It seems odd to me to worry about this, and in fact odd that visitors would both notice the absence of a TV and comment on it.
Data point: We don't have a TV in my student house, and no one has commented on that, even though we have had on average maybe 5 visitors per week for over a year. Offhand I'd estimate that about 50 unique people have seen our living room. I'm in Sweden.
Speculation: The common factor is you. You influence your visitors with regards to what topics to talk about and how to treat these topics.