[herewhere]

Around a decade ago, I was part of the team responsible for msnbot (a web crawler for bing). There used to be robot.txt (forgot the extension now). Most of the website was giving 10-20x higher limits to googlebot than rest other crawler.

Google definitely has unfair advantage there.

Bing and duckduckgo still provide very reasonable result with 10-20x less resources but not at par of google.