Many (not all) of the people labeling this position as racist are themselves the immigrants. If you legally prevent someone from immigrating to your country, then the ability of the would-be immigrant to call you racist in a way that matters is severely curtailed (since they're not currently in your country, they might well not even be doing so in a language you speak).
I've said this a million times, there is no free lunch with using immigration to paper over population decline. One of the main problems has to do with housing. A baby will require their own separate housing in ~20 years. An immigrant needs it _today_. It's irrefutable that bringing in 100K immigrants, illegal or otherwise, in a year will strain the housing market more than 100K babies. We used to have a sustainable way of keeping pace with population growth.
Indeed, the rate of new housing construction in the US declined substantially after the 2008 GFC and never recovered. We are now experiencing the result of that. You can’t print housing stock. Immigration quota should be a function of housing costs and availability, otherwise you’re bidding up the cost of housing for everyone for gains for some.
Subsidize the housing if you must to ensure you have affordable housing for peak population before decline kicks in, as it will everywhere.