Hacker News new | ask | show | jobs
by slipknotfan 1 day ago
https://sr.ht/robots.txt is a robots.txt
2 comments
gus_massa 8 hours ago
Ignoring that AI bot will ignore it, ...

Doesn't "User-agent: AwarioRssBot" needs it's own "Disallow: /" line?

link
zionsati 1 day ago
Do LLM crawlers adide by this?
link