| HN Mirror

Y	Hacker News new \| ask \| show \| jobs


	by frereubu 5 days ago
	This is good to know, but a bit of all-or-nothing. It's a shame that, for example, Google doesn't support the crawl-delay field so you can tailor their crawling to your setup: https://developers.google.com/crawling/docs/robots-txt/robot... I presume it would also cut you off even more from referral traffic.

1 comments

They don't support it yet. It seems as if Cloudflare is trying to use it's power to force Google to change this though.