[wilg]

If I were making a search engine or AI crawler, I would simply pose as Googlebot

[jsheard]

Google actually provides means of validating whether a request really came from them, so masquerading as Googlebot would probably backfire on you. I would expect the big CDNs to flag your IP address as malicious if you fail that check.

https://developers.google.com/search/docs/crawling-indexing/...

[wilg]

You could maybe still only follow robots.txt rules for Googlebot.