|
|
|
|
|
|
by pmdr
14 days ago
|
|
|
This. What even is the point of blocking scapers if Google consumes your content anyway and serves it as an AI answer? These are sad times we're living as far as openness of the web goes. People would have less of a scraping problem if their websites didn't ship with 20MB of JS. |
|
|
Google bot is generally fairly well behaved, but this is not the case for all scrapers and it can cause significant traffic (and expense).