[tjpnz]

At the risk of sounding flippant you might print your articles out onto sheets of paper and send them to interested parties by mail.

I'm sure a standard not unlike robots.txt will emerge. That might give some comfort, although I would remain sceptical given that many crawlers refuse to honour it.

[JohnFen]

> you might print your articles out onto sheets of paper and send them to interested parties by mail.

Or, better, do what I've actually done and make the websites private, invite-only.

> I would remain sceptical given that many crawlers refuse to honour it.

yeah, a robots.txt-like solution isn't adequate for just that reason. I don't rely on robots.txt alone to stop crawlers because of your observation here.