|
|
|
|
|
|
by kentonv
347 days ago
|
|
|
> There's no technical enforcement of the policies in the file, it's up to the client to honor them. That's incorrect. Cloudflare does in fact enforce this at a technical level. Cloudflare has been doing bot detection for years and can pretty reliably detect when bots are not following robots.txt and then block them. |
|
|