[ancientsofmumu]

I think it would be great if we had Code Forge index to search uniquely. In this index are only the myriad of code hosting sites around the internet - shared hosting like gitlab, github, sourcehut, sourceforge, codeberg, and all the project instances like the kernel.org, GNU Savannah, GNOME, KDE, BSD, etc. Probably hundreds of them out there, and allow people to submit their own self-hosted Gitea/Gitlab/sr.ht/etc. instances to be crawled - maybe even suggest a robots.txt entry your crawler could key in on as "yes please index me, hutbot".

[ksherlock]

Long ago -- 2006 to 2011 -- google had a functional source code search engine: https://en.wikipedia.org/wiki/Google_Code_Search

I don't recall if it supported SourceForge and GitHub (2008) but it certainly included gzipped tarballs which were popular and prevalent at the time.