Y
Hacker News
new
|
ask
|
show
|
jobs
by
jsheard
699 days ago
Google actually provides means of validating whether a request really came from them, so masquerading as Googlebot would probably backfire on you. I would expect the big CDNs to flag your IP address as malicious if you fail that check.
https://developers.google.com/search/docs/crawling-indexing/...
1 comments
wilg
699 days ago
You could maybe still only follow robots.txt rules for Googlebot.
link