Hacker News new | ask | show | jobs
by twoz 6184 days ago
Does Yahoo! not have a robots.txt file?

http://yahoo.com/robots.txt Sorry, the page you requested was not found.
1 comments
jdrock 6184 days ago
If that page doesn't exist, then according to specification, they don't and you can crawl any page.
link
twoz 6184 days ago
Update: Some Yahoo! subdomains have 'em.

Yes:

    http://search.yahoo.com/robots.txt
    http://groups.yahoo.com/robots.txt
    http://realestate.yahoo.com/robots.txt
No:

    http://maps.yahoo.com/robots.txt
    http://omg.yahoo.com/robots.txt
link