|
|
|
|
|
by flutas
805 days ago
|
|
> Except the fiist thing openai does is read robots.txt. Then they should see the "Disallow: /" line, which means they shouldn't crawl any links on the page (because even the homepage is disallowed). Which means they wouldn't follow any of the links to other subdomains. |
|