|
|
|
|
|
|
by JimDabell
347 days ago
|
|
|
> AI bots > You can opt into a managed rule that will block bots that we categorize as artificial intelligence (AI) crawlers (“AI Bots”) from visiting your website. Customers may choose to do this to prevent AI-related usage of their content, such as training large language models (LLM). > CCBot (Common Crawl) Common Crawl is not an AI bot: https://commoncrawl.org |
|
|