How are you forced into a conversation with a group of people you chose to go somewhere with? If you didn't want to talk with them, don't accept their invitation to get coffee
It is very rare that you really have to talk to someone, IME. Usually, it's more of a..."I don't really like or have a lot of common with this person but I've known them forever so what the heck"