Hacker News new | ask | show | jobs
by beejiu 336 days ago
That's surprising because Googlebot publishes IP ranges for its crawlers and it's fairly simple to block fake crawlers these days (super easy through Cloudflare, for example).
3 comments

doesn't google also run some "undercover" bots to verify that you don't serve very different versions of your website to users vs bots ?
In my experience 12ft.io was pretty much useless after a honeymoon period of a few months when it first came out so I wouldn't be surprised. The Googlebot method used to work with almost everything but at some point major news orgs caught on in quick succession and I gave up even bothering to try it.
this shouldn't work due to reverse DNS checks