Y
Hacker News
new
|
ask
|
show
|
jobs
by
gulbrandr
5005 days ago
Surprisingly the content of robots.txt is:
User-agent: * Disallow: /
http://webplatform.org/robots.txt
2 comments
wwwtyro
5005 days ago
I can't begin to fathom a motivation for that. Can anyone think of a reason, or is this likely simply a mistake?
link
criswell
5005 days ago
The site doesn't even seem close to being useful or complete yet. I think it's intentional.
link
acdha
5004 days ago
They didn't want search traffic until the content and structure start to stabilize? Google, et al. will crawl for years after a URL 404s so it might make sense to hold off until the first round of stabilization
link
desbest
5005 days ago
Maybe someone from Facebook made the robots.txt file, ayyy!
link
thenonsequitur
5000 days ago
The site was in super-alpha. I think it was intentional.
link
HendrikR
5004 days ago
This has already been changed to #User-agent: * #Disallow: /
link