|
|
|
|
|
|
by Anamon
48 days ago
|
|
|
Another possibility is that their crawling code is just that bad. A lot of these GenAI companies are dogfooding in their software development, and the quality can be seen whenever some of their code is released or leaked. It seems very plausible to me that their crawlers could simply barely functional, buggy messes. They have unlimited venture capital to burn, so it doesn't really matter if they scrape a site 100,000 times a day when once would have been enough. |
|
|