It's not elitist to like something that makes you happy, even if that thing is being around a group of wealthy, intelligent people. Elitism is about how you treat the other people. Being elitist would mean refusing to socialise with any group other than the one MIT crowd because they're too 'dumb' to be fun around. I don't think that's what jkestner is suggesting; that he found a group he's happy with doesn't mean he refuses contact with the other groups.
The desire to have a community and shared culture doesn't make one elitist. The GP could have as easily said they found bookish people they could relate to by becoming a librarian. Would that have sounded elitist or was the MIT name-dropping what upsets you?
"Elite" simply means "drawn out of the larger pool". Sure, there is a lot of social baggage around that word, as in a sense of superiority, but the in the general case there is no such implication.
What's wrong with hanging around with folks who get your jokes (and vice versa)? They don't have to form the totality of your identity.
Somehow it seems uncontroversial when someone finds a group of, say, fans of their favorite team (or rock band) and says, "now I am with my people!" Is there some peculiar difference in regards to books or maths jokes?
Is it elitist to want to be around other kids you can mentally relate to? I suffered in school until I transferred to a 7th grade public magnet school. For the first time in my life I felt like I belonged because I could be myself with the other kids, most of whom happened to be from poor immigrant families (it is not a class or culture issue).