|
|
|
|
|
|
by andrethegiant
593 days ago
|
|
|
[Shameless plug] I'm building a platform[1] that abides by robots.txt, crawl-delay directive, 429s, Retry-After response header, etc out of the box. Polite crawling behavior as a default + centralized caching would decongest the network and be better for website owners. [1] https://crawlspace.dev |
|
|