Hacker News new | ask | show | jobs
by pestaa 4367 days ago
Why wouldn't it consider the archived state of robots.txt?