This isn't like trying to start a new workout routine where the worst case scenario is some mild discomfort. The downside of contact with aliens is the obliteration of unique life forms and/or humanity.
If there are very threatening aliens, I would prefer to expand into the universe and have multiple inhabited planets and meet them being redundant rather than they just happen upon us and destroy us in a single go. Exploring the universe is more interesting anyhow.
In the history of human migration, there have always been some who have decided to stay behind: https://en.wikipedia.org/wiki/Early_human_migrations#/media/...