|
|
|
|
|
|
by fsniper
5069 days ago
|
|
|
That's it. A "bad bot" would not check robots.txt, but a legitimate user would check it. So looking for the software not checking robots.txt combined with user agent matching for good bots, you would have a good matching ratio. |
|
|