Hacker News new | ask | show | jobs
by Portoaj 237 days ago
Hi wibbily, your point on the first conflicts is fair enough, we'll update that.

I somewhat disagree with you on the last conflict: you have one document stating pretty clearly that returning a 503 for the robots.txt "blocks all crawling". The other document states there's a 12 hour block after which Google may decide to crawl the other pages (not just the robots.txt like you said).

Thanks for the feedback though, definitely some work to be done on validating the conflicts we surface.