Vertically up it's just too a higher local gravitational potential, no? Sure, we flip right-left when we look at things upside down, but that's a nuanced usage.
How do you use closer and farther to determine whether a magnetic field is pointing up or down? It's not like there are little arrows on magnetic field lines you can just look at.