Hacker News new | ask | show | jobs
by Chiron1991 366 days ago
That's why you can give guidance to crawlers using sitemaps.
1 comments
Quarrel 366 days ago
Which, just like robots.txt, worked decently well until the AI bot crawlers overwhelmed all else.
link