Hacker News new | ask | show | jobs
by slipknotfan 51 days ago
https://sr.ht/robots.txt is a robots.txt
2 comments
zionsati 51 days ago
Do LLM crawlers adide by this?
link
gus_massa 50 days ago
Ignoring that AI bot will ignore it, ...

Doesn't "User-agent: AwarioRssBot" needs it's own "Disallow: /" line?

link