That just shifts the problem to specifying the direction of the magnetic field, and you can't use the concept of left vs. right in that definition without creating a circular dependency.
Vertically up it's just too a higher local gravitational potential, no? Sure, we flip right-left when we look at things upside down, but that's a nuanced usage.
How do you use closer and farther to determine whether a magnetic field is pointing up or down? It's not like there are little arrows on magnetic field lines you can just look at.