Plenty of people choose sacrifice, hard work and discomfort because they find more fulfillment in it than idle leisure. Those who wish to remain behind can stay put, just don’t stand in the way.
Except that moonshots on the level of stellar diaspora require that sort of sacrifice from an entire society. If the 5% of the population that actually gave a shit about the idea of humans in the stars could do it by themselves I'd say go for it, but if it's going to happen they'll have to con people who will get nothing out of it into going along, which is obviously problematic.
It may only be 5% of people that care about space exploration by itself but a lot more will care if we’re lucky enough to only get a house sized impactor as a warning shot. At that size there’s a hope that there would only be severe property damage and no loss of life.
Chelyabinsk should have been that wake up call but most people don’t think about it deeply enough to realize how much worse it can be.
If we’re unlucky then the first big impact we experience will be our last.