Hacker News new | ask | show | jobs
by randombytes6869 2198 days ago
To those lamenting that they're scraping... Google is the biggest scraper of them all. Facebook, Amazon, Google, Microsoft. All the big boys scrape voraciously, yet try their best to block themselves from being scraped. Scraping is vital for the functionality of the internet. The narrative that scraping is evil is what big companies want you to think.

When you block small scrapers from your site but permit giants like Googlebot and Bing all you're doing is locking in a monopoly that's bad for everyone

1 comments

Google has the (often implicit) permission of the website owner to scrape. OTOH, Google Flights explicitly disallows scraping results.
No, Google's scraping is opt-out only, which they offer to be friendly.

Google does not need anyone's permission to scrape publicly accessible data, and they are not required to follow any opt-out requests.