[klenwell]

I would assume that any site that was implementing JS-level blocks also has the appropriate robots.txt file in place.

[greglindahl]

That's not true in the actual web, however.

The best example is a large number of unimportant sites that send 429 errors for /robots.txt if they think it's a scraper. A 4xx result for robots.txt is considered to mean no robots.txt for most crawlers. So the website is getting the reverse of what it thought it was getting.