Hacker News new | ask | show | jobs
by wseqyrku 40 days ago
The point of the post was that you need to add robots.txt (or similar) because it's a thing, and also tell us where they are.
1 comments

> add a robots.txt

Which bots will then ignore.

I didn't expect I'd get to use this so soon. Obligatory canned advisory:

https://www.felesatra.moe/blog/2026/06/19/robots

They still ignore it and scan everything even though you have told them not to scan certain paths.