[warkdarrior]

But there is: https://developers.google.com/search/docs/crawling-indexing/...

There is an user agent for search that you can control in robots.txt.

    user-agent: Googlebot

There is another user agent for AI training.

    user-agent: Google-Extended

[foota]

Wow, I had no idea this page existed, thanks for the reference!