It would take a proper user interaction study to find out, but I don't think that I am wrong. Imagine a photograph on a table that you want to see better (make larger). Your natural motion is to touch and pull, which is dragging down.
Hold control on your MBP and scroll up. It zooms in.
This is a rare interaction, which is arguably backwards.
command and '+' zooms text. Scrolling down generally scrolls to the end of a page. Scrolling down here scrolls to the end of the galaxy.
Those are all logical, not physical mappings. They are not relevant here.
Hold out your hand and make the "unpinch" gesture, which enlarges photos in iOS and Android. Which direction do your scrolling fingers move?
Left and right? You're seriously stretching to make your argument. People pull things towards them to get a better view. They push them away to see the bigger picture. Done.