[loa_in_]

We already have robots.txt in theory.

[layer8]

I didn’t think robots.txt would be applicable to URLs being copied around, but actually it might be, good point. Though again, collecting that robots.txt information could make it easier to search for such URLs.