New Yorkers love to hate Time Square but it's really not so terrible, just crowded. It's in close proximity to everything, and it looks pretty at night. And as you said, it's really easy to get to.
Yes, the Port Authority: aka the world's worst bus station. But you go a few blocks and you're in Hell's Kitchen which I find a pretty interesting neighborhood with nice ethnic restaurants. Or walk south 10 blocks or so into Chelsea.
When I'm in NYC I often stay at about 42nd and 10th out of choice even if I'm not at a Javits event. It's near things but is out of the craziness of Times Square itself.