|
|
|
|
|
|
by ikeboy
2464 days ago
|
|
|
if they're running on AWS, which most crawlers are not When I've run scraping software in the past I used DigitalOcean, which doesn't contain a requirement to abide by robots.txt. As far as I can tell it's both legal and consistent with their ToS to run a program that makes purchases on a website. |
|
|