[tomxor]

In light of the recent surge in scraped content trumping original content on Google search, this is so true... people who scrape your site do not care about your preferences in a txt file.

[stark98]

Yeah, I feel like it only discourages legitimate bots, but for malicious ones it's a big red sign saying "SENSITIVE CONTENT HERE, SCRAPE IT"