[jeroenhd]

You can on some devices (many Chinese brands, funnily enough) and on custom ROMs.

There are also (open source) firewall apps that will let you block (non-system) apps if you're on a stock ROM like me.

Technically, this is a permission, just not a user-grantable one. Google has moved quite a few permissions from inherent to user-grantable, but most apps don't work without internet (unfortunately) so I doubt they will do it for the internet permission in stock android.