Hacker News new | ask | show | jobs
by striking 3190 days ago
Yeah, people like to try and crawl Google. So they're keeping robots out.

But it's actually really nice to be able to pull a result or two from ddg by crawling, or to be able to use a VPN without having to solve a zillion captcha.

1 comments

I run into those captcha's occasionally and generally reconnecting to a different VPN makes them go away. I always assume they are tied to bot activity coming through the same IP or range. I've scraped google in the past (years and years ago) and from what I recall you end up hitting the captchas after a set number of results in too short a period. Back then it was a pretty standard SEO activity, but it's usefulness was minimized when google started localizing results.