Y
Hacker News
new
|
ask
|
show
|
jobs
by
hoistbypetard
77 days ago
If you do that and don't follow robots.txt, you are blocked. If you do that and follow robots.txt, fine. That's all we wanted you to do anyway. Just follow the instructions that well-behaved scrapers are meant to follow.