[npunt]

no I didn't see that in there, I just agree that robots.txt makes sense for web crawlers but not archival purposes