The fear stems from the replacement after Scalia -- if Ginsberg is replaced and Kennedy is no longer the median vote, the Court will take a sharp turn on a lot of social issues.
I agree that the Court will take a sharp turn on social issues. The question still that remains will be what cases could be brought before the Court that would actually give them the chance the overturn the previous rulings?
Typically the consequences of a Supreme Court ruling is that lower Courts resolve the cases in the direction of the Supreme Court's ruling and if you are foolish enough to appeal up to the Supreme Court, the Supreme Court simply declines to hear it.
There is nothing preventing the Supreme Court from deciding to hear an appeal (eg, of a Clerk who doesn't want to issue marriage licenses) and then issuing an sweeping decision completely to the opposite of the new status quo.
That is the risk of relying on the Supreme Court to recognize unenumerated rights; the recognition will only last as long as the Court remains favorable to the right.