Hacker News new | ask | show | jobs
by jedahan 283 days ago
My experience disagrees with the 'Respects robots.txt' column for most of the bots listed. Would love to see more details of how they determine that metric.
1 comments

Are you verifying the IP, or just blindly trusting the user agent?
Good question - I am just putting up robots.txt, and seeing little to no decrease in traffic. I have not tried verifying that server logs user agent corresponds to specific IP addresses. Do you have resources where all the AI bots post their list of IP addresses? Would be easier to just ban by IP completely. From what I've read these bots rotate and use residential blocks so I am not sure I can even see all of them.