Hacker News new | ask | show | jobs
by deadly_syn 1145 days ago
Robots.txt isnt a blocklist; more of polite suggestions you have to hope are respected.