Isn't that how it is in real life in general? If your work colleagues are duds, you're kind of on your own. At least traveling you can fall back on hostel friends, Couchsurfing or Meetup, or going out for a beer and meeting people who might be interested in a foreigner.
the main difference is length of interaction. With work/school the people in your social circle will be there for years, for better or worse. Friends made while travelling tend to be more ephemeral.