|
|
|
|
|
|
by reconnecting
122 days ago
|
|
|
Good question, at least OAI-SearchBot is hitting robots.txt. I assume the real issue is that what overloads the servers like security bots, SEO crawlers, and data companies — are the ones that don't respect robots.txt in full, but they wouldn't respect LLMs.txt either. |
|
|