Switzerland doesn't import unskilled workers directly, they just get them as refuges that they accept. They do not take them in as short term residents, that would be very unswiss.
The situation's changed post-Schengen, but Switzerland traditionally did have a guest-worker program, focused on short-term and seasonal workers who wouldn't gain any kind of rights. These were mostly from Europe, not refugees from elsewhere (the vast majority from Italy, with smaller numbers from Spain, Turkey, and Portugal). Under pressure, some were subsequently given residence rights, or (for those from EU countries) gained them later as a result of the Schengen treaty. See the section "Post-war labor migration" here: http://www.migrationpolicy.org/article/switzerland-faces-com...