Hacker News new | ask | show | jobs
by alecco 3251 days ago
Did you respect robots.txt?
2 comments
333c 3251 days ago
It sounds to me like an obvious no, if they have a large team to get around countermeasures.
link
rk06 3250 days ago
Considering the effort that went into it.

I am pretty sure crawling robot.txt links was their P1 requirement.

link