Hacker News new | ask | show | jobs
by Epskampie 448 days ago
In line 1 of of the linked page: "waste the resources of AI Crawlers and other bots that don’t respect “no crawl” directives".
1 comments
ccgreg 447 days ago
Does that indicate the robots.txt is how "no crawl" is indicated? robots.txt doesn't have "no crawl", it has allow and disallow.
link