[ryukoposting]

Off the top of my head, the most trivial approach is also trivial to defeat. Google sets up a VPN? Just give its IP address special treatment just like you're already doing with the geofencing trick.

The only thing I can think of that'd be tough to deal with is giving that VPN server a second route to the internet, and hiding that behind the biggest consumer CG-NAT you can find. They'll have to use a bogus UA string too.

To be fair, it's well within Google's capabilities. I'm surprised they don't already do something like this, honestly. Trivial? I wouldn't say so.

On a related note: between SEO abuse like this, and the ongoing systematic IP theft by OpenAI, Google, and friends, I worry we're entering an irreversible dark age for the web. Crazy abuses of NAT and UA strings will run rampant, and the only solution will be to serve nothing of value to anyone at all without a paywall.

[nisa]

Google has caching proxies in every country for almost every ISP. My rural ISP in Germany with only 65k users has it. I'm sure adding this to the contract is not impossible. On the other hand I have no idea how big the amount of data is that Google is moving around for crawling but for some statical sampling it might be enough to use the ASN of these boxes. On the other hand this probably amounts to more abuse Mail for the ISP. Lots of bandwidth that is more expensive for the ISP than Google and it's working directly against having these boxes there in the first place to remove traffic leaving the ISP.