Hacker News new | ask | show | jobs
by jl6 616 days ago
I have observed this bot requesting URLs that haven’t been live for over a decade, and to which no reference can now be found in search engines. I imagine there must be a private trade in URL lists.
1 comments

maybe they are using commoncrawl, webarchive, yandex as indexes?
In addition to those it's also possible they just found a website that published a scraped list back then and got de-indexed for obvious spammy content.

I would not be surprised if there are still some auto generated link directories left from the "golden ages" of blackhat.