[kibwen]

What makes the latter sentence sound highfalutin is that you've been required to contort it away from idiomatic American English sentence structure in order to force in a "whom". The usual way of phrasing the sentence avoids "who" entirely: "If the people I speak with" or "If the people I'm talking to".

[lupire]

Who(m) are you talking about?

It's the "m", not the sentence structure. The awkward archaic structure is a different, often comorbid, affectation among Latinophiles.