> To run a computer cooler than that would require a heat pump, which means adding additional energy to the system than what is needed for our computation.
That does not address it. It is true that heat has to be removed to maintain the lower temperature. But the rate at which this has to be done depends on heat generation and quality of isolation. If temperature inside is 1e-100 K, energy cost of only bit flipping becomes negligible. Energy cost of refrigeration depends then on how good the isolation is. With better and better isolation, the cost goes down, the only limit is zero.