[Aurornis]

All of the crawlers present themselves as being from one of the major companies, even if they’re not.

Setting user-agent headers is easy.

[cyanydeez]

at the same time, all the AI providers have some kind of web based AI agent, so let snot pretend they're crafting their services in care of other peoples websites.

[CaptainFever]

I highly doubt that people are using AI agent features so frequently and so concentrated-ly that it brings down websites.