[pessimizer]

"You're difficult to talk to" is not a hostile statement. It's also what you say to someone after they're being difficult, not before. I don't think the reply of "no, you" would be the problem.

And if somebody tells me I'm difficult to talk to (which has happened about particular subjects with particular people), my response is not to look for something to accuse them of so they don't win. I either ask why, or if I know I'm being reasonable, terminate the conversation.

Talking to somebody who is constantly trying to equalize their position with you is fine usually, but trying to criticize, teach, or suggest a different approach to them is impossible. They don't listen, immediately get angry, and try to find something they can criticize about you (even if they don't really mean it.) I think it's a sign of abuse.