Hacker News new | ask | show | jobs
by rezbull 1563 days ago
“…treat the absence of one as a lack of consent. ” - do you have a source for this?

Their documentation states otherwise https://developers.google.com/search/docs/advanced/robots/ro...

1 comments

You are correct; my mistake. Nevertheless S3 returns a 403 (unauthorized) response for robots.txt by default which causes Google not to index it.