[mikestew]

like an exotic word that people only pull out to be semi-fancy, like "whom"

Semi-fancy? Man, that's a pretty low bar for fifty-cent words. I use it so I sound like I actually went to school and paid attention. If those with whom I speak find basic grammar fancy, that's on them.

[kibwen]

What makes the latter sentence sound highfalutin is that you've been required to contort it away from idiomatic American English sentence structure in order to force in a "whom". The usual way of phrasing the sentence avoids "who" entirely: "If the people I speak with" or "If the people I'm talking to".

[lupire]

Who(m) are you talking about?

It's the "m", not the sentence structure. The awkward archaic structure is a different, often comorbid, affectation among Latinophiles.