Hacker News new | ask | show | jobs
by squarefoot 1154 days ago
"No, I do not honor robots.txt files when scraping data."
1 comments

Wondering if there's a list of IPs or a reliable technique to block them else our hosting bills will be inflated by script kiddies running their crawlers. Curious how much extra it's costing us to have them keep sucking images, audio, video and text over and over.