...but not to download. You can clone a repo and download release artifacts without a PAT. That's only necessary for interacting with the API for actions that need authentication, which would be anything involving mutating a repository.
GitHub access tokens are a bit of a nightmare since you can't limit the permissions for a token. Only workaround I've found is to create another GitHub user for an access token and restrict that user's access.
> You need an access token to publish, install, and delete packages in GitHub Packages.
https://docs.github.com/en/packages/using-github-packages-wi...