The cat is making it clear that you are not respecting its personal space. (It can't talk, yeah?) If you never interact with it, it has no idea whether or not you're cool.