[staplung]

If they can't hit `/*/tree` is there a way to know the URLs of the files?

[pancrufty]

Direct links from crawlable pages

[kadoban]

Sure, clone the git repo.

[utopcell]

GitHub would not be happy with Google cloning all repos, and many of them at a high frequency, in order to circumvent a robots.txt restriction.

[kadoban]

They're clever people, they could just do partial updates (pull instead of clone). I doubt it'd be that much of a strain.