[delfinom]

>One way Google maintains their monopoly is that many websites block all bots except for the Google indexing bot.

Because Googlebot is actually _friendly_.

O boy are there bad bots out there for search engines that often DDOS your site to shit.

Yandex is probably the worst out there from experience. Not only will it hammer you, they put in extremely aggressive retries such that even if you ban it temporarily for 4 hours, the literal second your IP block of their spiders expire, they will flood you with requests, and they don't even know how long the ban is, their bots just keep trying and trying.