Hacker News new | ask | show | jobs
by kortilla 2547 days ago
If you don’t want something public, don’t allow a crawler to find it or access it. The people you want to hide stuff from are just going to use search engines that ignore robots.txt
1 comments

If you don't want someone or a bot to find it, don't put it online.