Hacker News new | ask | show | jobs
by rk06 3254 days ago
Considering the effort that went into it.

I am pretty sure crawling robot.txt links was their P1 requirement.