[lp0_on_fire]

Being only tangentially familiar with the indexing bots…what legal barriers are in place to prevent a competitor from impersonating a google indexing bot? Is it just a matter of the google bot originates from x subnet so that’s the only one webmasters allow? What’s to stop a competitor from running their own bot but sending user-agent: totally-the-google-indexing-bot-and-not-a-competitor?

[jiggawatts]

Source addresses: the Google bot traffic comes from a small set of Google-owned IP address blocks.

Third parties bake this into things like Web Application Firewall (WAF) rules. For example, Azure App Gateway WAF has a policy category for “known bots” which includes Google but excludes your tiny AI startup.

It’s a moat built by giant corporations to keep tiny players in their place.

Google "helpfully" publishes their bot source IP addresses: https://developers.google.com/static/search/apis/ipranges/go...

AWS also provides named rules such as "bot:name:googlebot": https://docs.aws.amazon.com/waf/latest/developerguide/aws-ma...

[creato]

I agree it’s a moat, but why would azure restrain competition from google? I think it’s just yet another example of an anti-abuse collateral damage, like email anti-spam blocking small unknown servers or Cloudflare blocking unknown IPs.

[sokoloff]

Because Azure customers want Google to be able to index their sites.

Would you host your e-commerce or social media site on a cloud provider that blocked Googlebot?

It's not Microsoft giving Google a handout out of the goodness of their hearts; it's Azure customers demanding that functionality. (Those Azure customers also don't care about a random little search startup and probably don't want to pay any egress fees to serve traffic to it.)

AWS offers the same: https://docs.aws.amazon.com/waf/latest/developerguide/waf-bo...

Oracle as well: https://docs.oracle.com/en-us/iaas/Content/WAF/Bot/good_bot_...

Akamai as well: https://www.akamai.com/products/bot-manager

[whatshisface]

Moats are anticompetitive, that's the entire idea behind why founders want them. Who else would a moat keep away, customers? ;)

[kazinator]

Customers are kept in by moats.

[whatshisface]

That's lock-in, moats are things that keep your competitors out. Source: https://www.investopedia.com/ask/answers/05/economicmoat.asp

[nova22033]

For example, Azure App Gateway WAF has a policy category for “known bots”

So why should we take Satya's testimony seriously? Azure is a Microsoft product and if what you're saying is true, Microsoft is part of the problem.

[sokoloff]

Almost surely that Azure customers want that.

https://news.ycombinator.com/item?id=37746480