|
|
|
|
|
|
by n_u
248 days ago
|
|
|
Respecting robots.txt is a convention not enforced by anything so yes the bot is certainly free to ignore it. But I’m not sure I understand your distinction. A scraper is a crawler regardless of whether it is “custom”or an off the shelf solution. The author also said the bot identifed itself as a crawler > Mozilla/5.0 (compatible; crawler) |
|
|