[Laremere]

Two other points to consider:

Atmosphere is slowing you down as you go up as well, so lift off requires fighting that as well. Therefore the energy you've expended is not entirely stored in potential energy. (Fun fact: If it weren't for the atmosphere, rockets would actually take off almost horizontally. They go up at first to get out of the thickest air before going sideways.)

The craft is much lighter, because it no longer has the payload, and also has used almost all of its fuel.