Hacker News new | ask | show | jobs
by 112233 102 days ago
Just make valid robots.txt and sitemap.xml, please, so I can crawl and update mirrors of the sites I am interested in with least amount of impact on the site.