Port scanning was always considered abuse until researchers decided they needed additional blog post material. I worked for a “business ISP” in the early 90s and we had no qualms kicking someone off the network for it, although usually the originator of the scan had themselves been compromised… because… as I say it was considered at least a breach of terms of service.
I ran engineering for what was at the time the most popular ISP in Chicago from 1995-1997 (the company later sold to RCN) and port scanning has never been considered abuse, though there have always been people who wished it was and loudly proclaimed it so.
I'd say that port scanning is attack prep. When you port scan me I take an adversarial stance. Certainly volume plays a role in determining whether scanning is abusive or not.
There are differences between walking by and noticing a door, knocking on a door, trying a key in a door, trying to defeat the lock. Before you can break the lock you have to notice the door. I haven't met any friendly door knockers on the internet.