Hacker News new | ask | show | jobs
by yakubin 1401 days ago
The web is a public space. There are no doors.

Robots.txt is a way of saying "these are not the pages you are looking for".