Y
Hacker News
new
|
ask
|
show
|
jobs
by
wseqyrku
40 days ago
The point of the post was that you need to add robots.txt (or similar) because it's a thing, and also tell us where they are.
1 comments
Geezus_42
40 days ago
> add a robots.txt
Which bots will then ignore.
link
Ferret7446
39 days ago
I didn't expect I'd get to use this so soon. Obligatory canned advisory:
https://www.felesatra.moe/blog/2026/06/19/robots
link
Geezus_42
39 days ago
They still ignore it and scan everything even though you have told them not to scan certain paths.
link
Which bots will then ignore.