[SamFold]

“With whom were you speaking” IS grammatically correct though. “Whom” is the object of the preposition “with”.

[Tyrannosaur]

Even if you want to differentiate "who" and "whom", "Whom were you speaking with?" is more natural.