|
|
|
|
|
|
by marginalia_nu
1165 days ago
|
|
|
Haha, nice! Crawler traps are a quite old phenomenon. Been around since before Google. Dunno about the others, but my crawler has a set depth it will crawl. It'll BFS for like 1000-10000 documents depending on some factors. |
|
|