Public Github with CI means the binary was built unmodified from source. You can turn off issues/PR and push only release branches with squashed commits.
Assuming you trust Github, of course. I think if someone is seriously worried code has been altered between source and maintainer-provided binary, his big concern will be the time it takes to audit the source code (which he also shouldn't trust). The build time will be inconsequential next to that.