Hacker News new | ask | show | jobs
by JohnFen 974 days ago
Nobody actually has to comply with robots.txt.