The concept of privilege is that you have time/energy/resources to do things like play chess instead of looking for work. I'm not sure if you're questioning the concept of privilege (which is fine) or saying that it specifically doesn't have anything to do with chess.