Hacker News new | ask | show | jobs
by littlecranky67 659 days ago
Well, there never was any law that required robots.txt to be honored. Big players like Google do, but I am not aware of any consequences if they wouldnt (of course UNTIL it is then regulated).