[haiku2077]

Either that or Anubis (https://anubis.techaro.lol/docs), yes.

[roguecoder]

So these companies broke the internet

[haiku2077]

Which companies?

OpenAI, Anthropic, Google? No, their bots are pretty well behaved.

The smaller AI companies deploying bots that don't respect any reasonable rate limits and are scraping the same static pages thousands of times an hour? Yup

[e3bc54b2]

Anecdote, but at least for tiny little server hosting single public repository, none of these companies had 'well behaved' bots. It may be possible that they learned to behave better but I wouldn't know since my only possible recourse was to blacklist them all AND take the repo private.

[haiku2077]

Those are the small companies spoofing their user agent as the big companies to dodge countermeasures.