Yeah, that was what I thought when I first worked with these APIs! But when you use PhotoKit, you have to explicitly opt-in to downloading files from iCloud.
AFAICT, PHAsset is only metadata. When I'm downloading the full-sized images, I use PHImageManager.requestImage() and pass in the PHAsset I'm looking at [1][2]. I know there's something similar for video, but I've never used it.
You can control the behaviour by passing a PHImageRequestOptions instance. This includes an isNetworkAccessAllowed bool which controls where Photos.app will download the file from iCloud if not present locally, and it defaults to false.