Hacker News new | ask | show | jobs
by trekz 718 days ago
Also note, Cloudflare blocks IPs of legitimate automated "good bots" by default, like RSS feed fetchers (1), but allows traffic to some of the most aggressive bots on the web, like Google bot and all sorts of AI bots (2).

(1) https://openrss.org/issue/144

(2) https://radar.cloudflare.com/traffic/verified-bots - GoogleBot, GPTBot, and AmazonBot are some of the first ones listed.
1 comments
theanonymousone 718 days ago
Blocking RSS fetchers which are "bots" by design, intent, and definition, is really bizarre!

Seems the poor indy bots scraping price pages once a week are totally out of lock then.

link