Hacker News new | ask | show | jobs
by gumballindie 1144 days ago
Does it honour robots.txt when scraping data?
1 comments

"No, I do not honor robots.txt files when scraping data."
Wondering if there's a list of IPs or a reliable technique to block them else our hosting bills will be inflated by script kiddies running their crawlers. Curious how much extra it's costing us to have them keep sucking images, audio, video and text over and over.