I recently un-denylisted AI crawler UAs from my blog's robots.txt, for several reasons:
- My blog is static content and it costs me ~nothing to serve the requests.
- The bots were ignoring robots.txt anyway.
- If there's ultimately a human driving the bot (e.g. someone asking "summarise this article"), I don't mind.
- It's like trying to block search engines. Just as I want my blog to turn up in search results, I want agents etc. to know it exists, too.
My original motivation for denylisting, years ago, was that LLMs were simply not very good, so training-set scrapers seemed like all downside with no upside.
- My blog is static content and it costs me ~nothing to serve the requests.
- The bots were ignoring robots.txt anyway.
- If there's ultimately a human driving the bot (e.g. someone asking "summarise this article"), I don't mind.
- It's like trying to block search engines. Just as I want my blog to turn up in search results, I want agents etc. to know it exists, too.
My original motivation for denylisting, years ago, was that LLMs were simply not very good, so training-set scrapers seemed like all downside with no upside.