Y
Hacker News
new
|
ask
|
show
|
jobs
by
randomdata
4565 days ago
Depends on the intent. If it is user-initiated (like say a mobile formatted version of the site), it wouldn't have to be obey the robots.txt, since it is not a crawler, just another web browser.