Hacker News new | ask | show | jobs
by phoronixrly 81 days ago
Well-behaved agents will obey robots.txt and not fall into the trap.