I tried to find a better way to do that, but public_repo is the least granular permission that github provides.