Hacker News new | ask | show | jobs
by adrian17 14 days ago
> After 15 minutes of confusion, it turned out Cloudflare had put a crazy robots.txt on my site without my consent (Cloudflare, love you guys, but this needs to stop).

Might be the first time I see someone complain about their website being protected from a scraper, instead of the other way around.
4 comments
sva_ 14 days ago
I am pretty sure you have to enable cloudflare to manage your robots.txt, it shouldn't be doing that by itself. Maybe they did it by accident, it is just 1 click.
link
zx76 14 days ago
My understanding is that the current default allows AI training bots - but this actually going to change in 2 months time.

https://developers.cloudflare.com/changelog/post/2026-07-01-...

From Sept 15 all new sites added to CF will even block Googlebot by default on any page that serves ads as I understand it.

I think it's CF trying to force Google to separate out their bot traffic into bots for training and bots for the search index.

I think CF sees a big opportunity to get businesses to pay them to allow certain uses of their data but block others.

They're also starting a registry of "Approved" crawlers.

link
nibbleyou 14 days ago
I think the issue is the lack of consent. Whether a service I use is protecting my website from scrapers or feeding everything to scrapers, some of us would prefer that it takes our informed consent before doing so.
link
dannyw 14 days ago
Cloudflare is explicitly a service for dropping requests, whether it’s DDoS attacks, as a WAF, or AI crawlers. It offers a lot more too, but this isn’t Cloudflare overstepping imo.

FWIW, I just set up a domain last week, and the web UI asked if I want to block AI crawlers or not.

Perhaps OP set it up agentically, and the agent didn’t pass an optional param correctly, or ticked the box for him?

link
bfjvibybd6cuvu6 14 days ago
It does.
link
el_io 14 days ago
You have to enable this, or atleast was the case when I tried less than 1 month ago.
link
dannyw 14 days ago
You still have to enable this as of one week ago. I suspect OP might have agentically set up a new Cloudflare domain, and who knows what the agent did during onboarding.
link
rgrieselhuber 14 days ago
It should still be done with consent. Robots.txt files don’t protect against determined scrapers anyway.
link
adrian17 14 days ago
I thought bot protection was one of reasons for using Cloudflare in the first place (next to general CDN hosting)? After all, they do show a captcha-like challenge on some websites, so I thought that even without robots.txt, it still would have prevented the automated request by default.
link