TL;DR: The two scenarios are not symmetric, the person in the spaceship undergoes acceleration whereas the observer staying behind does not.