Hacker News new | ask | show | jobs
by gerdesj 435 days ago
robots.txt is famously ignored by many ... agents ... out of pure spite.

That's a human thing.