Hacker News new | ask | show | jobs
by mxpxrocks10 2420 days ago
Some other notes: 1) you're not checking subdomains like blog.company.com or paths like company.com/blog 2) if you use something like zgrab you can do 1M site crawl in a couple of hours. Consider checking it out.