Yes. We can cap non-EU/EFTA immigration to zero but that's relatively small anyway. Getting out of Schengen-Dublin and more importantly the Freedom of Movement of workers would basically unravel all bilateral agreements.
EU citizens can freely live and work in Switzerland and vice-versa. It would be difficult to reliably cap immigration from other EU countries and stay in the Schengen Area.
When I was at CERN, it was before Schengen became a thing, so as Portuguese I had the same VISA issues as someone else coming out of the other side of the planet.
Worse, being at CERN wasn't a plus for the hiring process, I would need to apply to the position as if still living in Portugal, as my VISA was tied to CERN directly with a three month deadline to leave Switzerland after the contract duration.
It also did not help, that my fellow country folks do not have a positive image across the country, for various kind of reasons, which is another issue I experienced while living there, like being refused entry in clubs when showing a Portuguese ID card.
Eventually I moved back to another EU country, still I do visit Switzerland, from time to time.
Pity that right wing movements are taking off all over the place.
You're throwing a lot of words that you don't understand nor have much relevance to the topic.
Before bilateral agreements and the freedom of movement, not Schengen which was ratified much later and is completely irrelevant here, you needed a work permit, not a visa (lowercase), which anyway at CERN is the equivalent of a diplomatic permit given to all international and tax-exempt NGOs in Geneva/Switzerland. And of course you lose your CDL permit quickly after your contract expires.
Getting a B permit before FoM would specifically not have been as hard for you as for someone from another continent.