[nl]

This isn't true. Search engines rely on fair use: https://www.everycrsreport.com/reports/RL33810.html

[schemescape]

Agreed. Sites like GitHub actively monitor for unwanted crawling/scraping and throttle requests.