|
|
|
|
|
|
by l1n
694 days ago
|
|
|
> Sites use robots.txt to tell well-behaved web crawlers what data is up for grabs and what data is off limits. Anthropic ignores it and takes your data anyway. That’s even if you’ve updated your robots.txt with the latest configuration details for Anthropic. [404 Media] doesn't seem supported by the citation, https://www.404media.co/websites-are-blocking-the-wrong-ai-s... |
|
|