Why do you need to replace population? Is Japan really that bad of a place? And Japan's population has been in the 120-130 million range for the last 35 years, so it's not like there's been a huge swing either way.
Japan has had issues with finding workers to support their elderly population. No, it isn't a bad place. But yes, there are issues to solve that come with an aging population and a shrinking population.
And yet Japan has the second-longest life expectancy in the world. The "we need open borders or we'll end up like Japan" argument just doesn't seem very strong when Japan does well in most quality-of-life metrics.