|
|
|
|
|
|
by asdkjlfkasjdf
1137 days ago
|
|
|
exactly, but nobody in their sane minds would be using robots.txt to block crawlers as that is SEO (organic traffic from searches) disaster. People could be adding a specific robot user-agent, if they knew openai even existed before yesterday and was stealing their content. but nobody did. |
|
|