[cudgy]

The people that need to be at the harbor do not need offices … they are working on the docks or on the ships.

The people at the mineral deposits do not need offices either … they are working in the mines or operating equipment.

The vast majority of people that need offices for work can work almost anywhere.

[thaumasiotes]

That's largely true for the mineral deposits, but not at all for the harbors. Harbors necessarily host warehouses, tax assessors, harbormasters, lighthouses, repair services... and all of those require offices on site (though the lighthouse is capable of containing its own office).

(Offices are of course required at a mine too, but a comparatively tiny number of them.)