[svelle]

It's funny how every time this topic comes up someone says "I've had this happen and x is the worst" With x being any of the big AI providers. Just a couple minutes ago I read the same in another thread and it was Anthropic. A couple weeks back it was Meta.

My conclusion is that they're all equally terrible then.

[Aurornis]

All of the crawlers present themselves as being from one of the major companies, even if they’re not.

Setting user-agent headers is easy.

[cyanydeez]

at the same time, all the AI providers have some kind of web based AI agent, so let snot pretend they're crafting their services in care of other peoples websites.

[CaptainFever]

I highly doubt that people are using AI agent features so frequently and so concentrated-ly that it brings down websites.

[ipaddr]

I agree they are all creating negative value for site owners. In my personal experience this week blocking Amazon solved my server overload issue.