Hacker News new | ask | show | jobs
by brlewis 2536 days ago
It would even be a million times better to place the sensitive files inside /TOP_SECRET_FOLDER and disallow the entire path, avoiding to explicitly name the paths at least.

This is the only way to use robots.txt for semi-sensitive info, and obviously not for info so sensitive that it would be awful for it to get out. URLs can leak through proxy logs and shared browser history.