I think you misunderstood. The point is that it's not in his head - he knows his Indian colleagues are trying to figure out his caste. Non-Indians don't have a caste so he knows it's safe in that situation.
Thats what I am having trouble understanding. how does he know? Unless you take into account the context of them being Indian and his experience in India, they won't be able to tell whether its to figure out his caste, or just to have fun.
Have you ever been discriminated against? And on a consistent and ongoing basis? When people are after you all the time, you start to learn the warning signals.
I understand the mental stress of having people after you all the time, but that doesn't mean that all the people are after you all the time. In this case they may have extended a genuine invitation. And given the historic context, it may have led to him feel like it was a casteist action, that doesn't make his colleagues casteist.
It’s true that they may have extended him a genuine invitation. It’s also true that he may have overthought it. But that is kind of the point of the story, no?
After you endure enough discrimination and persecution, your trust in others are eroded. You start to irrationally question the motives of others. Eventually you lash out preemptively because you’re afraid.
This is why discrimination and oppression is bad. It tears at the fabric of society.