Hacker News new | ask | show | jobs
by abraham 5137 days ago
Where are you getting that? It doesn't match what I'm seeing. http://googleblog.blogspot.com/robots.txt
1 comments

Googled for "blogspot", picked first random domain I saw, "weliveyoung.blogspot.com", fetched "weliveyoung.blogspot.com/robots.txt" with curl, got a redirect to "weliveyoung.blogspot.co.uk/robots.txt", fetched that, voila.

Perhaps there is a user setting that controls it.

Looks like you can use whatever you want. The one I linked to is the default.

https://support.google.com/blogger/bin/answer.py?hl=en&a...