[yakubin]

The web is a public space. There are no doors.

Robots.txt is a way of saying "these are not the pages you are looking for".