[JohnFen]

> you might print your articles out onto sheets of paper and send them to interested parties by mail.

Or, better, do what I've actually done and make the websites private, invite-only.

> I would remain sceptical given that many crawlers refuse to honour it.

yeah, a robots.txt-like solution isn't adequate for just that reason. I don't rely on robots.txt alone to stop crawlers because of your observation here.