|
|
|
|
|
|
by Max-Ganz-II
241 days ago
|
|
|
The instant I dropped this on HN, gptbot was scanning the site. (Updated robots.txt, this site https://robotstxt.com/ai to the rescue. I had to update robots.txt on the Amazon Redshift site fairly recently - I had gone from something like 10mb of log file per day to something like 50mb of log file per day, from LLM crawlers.) |
|
|