[AnthonyMouse]

The irony here is that the people blocking all other crawlers are the ones shoring up their monopoly. If you can't block Googlebot because you need the search traffic but you block everybody else so that nobody other than Google can index your site, how do you expect to ever get any search traffic that isn't from Google?

[tempest_]

Search traffic is nose diving due to LLM use. The fundamental calculus with google is you install GA and it helps your SEO has changed and google is riding out what will eventually wither as people start to reevaluate the trade off.

[AnthonyMouse]

It's nose diving due to Google putting the LLM answer box at the top of the search results. But then isn't it even worse to be blocking every search engine crawler that isn't doing that?

[dannyw]

Even companies like OpenAI segregate their training crawlers (GPTBot for general corpus training; OAI-SearchBot for their search index inclusion; and ChatGPT-User for user-driven agentic browser use, etc).

When even OpenAI is more respectful of intellectual property and website owner control than you, that's a problme.

[AnthonyMouse]

> Even companies like OpenAI segregate their training crawlers (GPTBot for general corpus training; OAI-SearchBot for their search index inclusion; and ChatGPT-User for user-driven agentic browser use, etc).

They have three different user agent strings, anyway. The problem is obviously that you can't tell what someone does with data after you give it to them.

You also don't know that some third party isn't crawling the web with the user agent string "OAI-SearchBot" and then using the results for training or selling the data to the likes of Anthropic or OpenAI without telling them that.

In general attempting to use the user agent string for access control is not going to work and it seems like Google is being the less disingenuous party on this one by not blowing smoke.

[xnx]

How does Google Analytics "help your SEO"?