|
|
|
|
|
|
by FridgeSeal
428 days ago
|
|
|
It seems a good chunk of the issue with these modern LLM scrapers is that they are doing _none_ of the normal “sane” things. Caching content, respecting rate limits, using sitemaps, bothering to track explore depth properly, etc. |
|
|