[tompec]

It does respect robots.txt when crawling. I'll add more details about this in the docs.

[pryelluw]

I appreciate the reply. As someone who runs multiple CMSs it’s painful to deal with the ai crawlers these days. Specially the ones that don’t respect my terms.