Y
Hacker News
new
|
ask
|
show
|
jobs
by
rezbull
1563 days ago
“…treat the absence of one as a lack of consent. ” - do you have a source for this?
Their documentation states otherwise
https://developers.google.com/search/docs/advanced/robots/ro...
1 comments
otterley
1563 days ago
You are correct; my mistake. Nevertheless S3 returns a 403 (unauthorized) response for robots.txt by default which causes Google not to index it.
link