It seems like a bad comparison, since you don’t need to mine gold just to move it around. Transferring ownership can be accomplished for the amount of energy it takes to send a small electronic message.
From some quick searching around, volume on the exchanges is around 500,000/day while volume on the blockchain is over a million. If that’s accurate (and it may not be, I’m not an expert on this and the sources I found might suck) then most volume is on the blockchain by a pretty good margin.
I don’t know how much gold is actually shifted around physically, but the trading volume in a year is several times the actual amount of gold that has been mined, so I suspect the proportion of that which involves physical movement is tiny.