Indeed. My biggest bugbear with this is google's "location accuracy" service, which implicitly sends them not only your location, but a detailed map of any networks around you. It takes active effort to keep it turned off (and doing so actually breaks quite a few UI flows in google maps!), and still I've accidentally turned it on a few times and only noticed because I actively check for it. There is no reasonable definition of 'consent' which can be derived from that setting.
I never took that phrase to be reserved for sexual situations - in fact my mom used to use it with me all the time over toys, ice cream, etc. "Come ON, PLEEEASE?!" "No means no..."
I think it's a phrase that is often applied in sexual situations, but not intended to be exclusive to them.