[funnyflamigo]

> Seriously, I hate scrapers. I hate the people who make scrapers. I hate their lack of ethics. Fuck those guys.

Wait till you find out what half of Google's business is based on (spoiler - scraping).

I really don't think scraping itself is an issue 90% of the time. It's the behavior of the out of control scrapers that are the problem. A well behaved scraper should barely be noticeable, if at all.

[jjeaff]

At least google's scraping does result in your website being discoverable by users. So you get something out of it. That's not to say that sometimes Google is missing or stealing data they scrape. But at least there is some benefit. Many other scrapers are merely taking the data to compete.

[funnyflamigo]

I strongly feel that if a human can get to it manually, we have to accept that either it will be botted or humans will be paid to do it by hand (They call these people "analysts" or "market researchers").

I might argue that what google actually uses their scraped data for is their search engine - which is private. They simply allow us access to specially crafted queries, which they can and do manipulate (for many reasons, some good some bad).

The only thing I'd say meets that definition would be like Common Crawl.

[RobSm]

Exactly. I am surprised that the 'devs' can't figure out a way to block only annoying/excessive scrapers. Most likely they are just lazy and then just put 3rd party 'solution' and job done. Pay me.