|
|
|
|
|
|
by mellosouls
1489 days ago
|
|
|
The ethical argument has nothing to do with hiring hundreds of people to circumvent robots.txt requirements! The whole point is to avoid unnecessary or excessive crawling by bots that are engineered with no concern for anything other than the owners motivations, presumably financial in most cases. |
|
|