Y
Hacker News
new
|
ask
|
show
|
jobs
by
yamural
4888 days ago
There is no legal restriction for crawling any web site. But you should respect robot.txt document for every single web site. Such as:
https://www.facebook.com/robots.txt
http://web.archive.org/robots.txt