[regularfry]

How are the crawlers identifying themselves? If it's user agent strings then they can be faked. If it's cryptographically secured then you create a situation where newcomers can't get into the market.

[what]

Google publishes the ip addresses that google bot uses. If someone claims to be google bot but is not from one of those addresses, it’s a fake.

[regularfry]

And in that case both systems end up with a situation new entrants can't enter.

[usefulcat]

I don't see how that helps the case where the UA looks like a normal browser and the source IP looks residential.

[lacksconfidence]

How about if they claim to be google chrome running on windows 11, from a residential IP address? Is that a human or an AI bot?