Y
Hacker News
new
|
ask
|
show
|
jobs
by
mxpxrocks10
2420 days ago
Some other notes: 1) you're not checking subdomains like blog.company.com or paths like company.com/blog 2) if you use something like zgrab you can do 1M site crawl in a couple of hours. Consider checking it out.