|
|
|
|
|
by gbalduzzi
330 days ago
|
|
I think it depends on the definition of bad. I always consider "good" a bot that doesn't disguise itself and follows the robots.txt rules. I may not consider good the final intent of the bot or the company behind it, but the crawler behaviour is fundamentally good. Especially considering the fact that it is super easy to disguise a crawler and not follow the robots conventions |
|