Hacker News new | ask | show | jobs
by Gigachad 1460 days ago
“Private links” should be covered by robots.txt. The only case I see this happening is for those “anyone with link” shares and those are easy to cover.
1 comments

AFAIK, most of those "private" links are actually just "unlisted" but they're still public. I'm sure Bing is indexing those too...
Anything private should ideally be put behind authentication. If that isn't possible, than robots.txt. Search engines are _meant_ to index everything that is publicly accessible and not blacklisted on robots.txt.