Hacker News new | ask | show | jobs
by charcircuit 104 days ago
>Honors robots.txt

Is it possible to ignore robot.txt in the case the crawl was triggered by a human?