The argument I've heard for T-shirt sizes is that if you go to numbers people try to add them together when that's just not how it works. I do agree that T-shirt sizes don't work that well though.
I see what you mean about trying to think abstractly instead of about numbers. But once you have t-shirt sizes for some tasks, what do you do with them? You can't compare them. You can't convert them to date forecasts. You can't use them for sprint capacity planning like you can with story points.