You're not actually downloading the package from Packagist. It acts only as a metadata repository that points you to a source repository. For the others, I would assume they act as both a metadata and a source repository.
It takes a bit of work to store versions and manage them as a central repo. Composer will fetch the sources, as a tarball from GitHub if possible, and cache them locally. They are skipping being the central store and deferring to root sources.
They removed a lot of complexity by doing that and I bet more than a few people didn't notice.