[btown]

The headline is somewhat misleading: sites using Cloudflare now have an opt-in option to quickly block all AI bots, but it won't be turned on by default for sites using Cloudflare.

The idea that Cloudflare could do the latter at the sole discretion of its leadership, though, is indicative of the level of power Cloudflare holds.

[GrayShade]

> sites using Cloudflare now have an opt-in option to quickly block all AI bots, but it won't be turned on by default for sites using Cloudflare

Do you have a source for that? https://blog.cloudflare.com/content-independence-day-no-ai-c... does say "changing the default".

[mattcollins]

"This feature is available to all customers, meaning anyone can enable this today from the Cloudflare dashboard."

https://blog.cloudflare.com/control-content-use-for-ai-train...

[bitpush]

It is now an adversarial relationship between aibots and website, and cloudflare is merely reacting to it.

Would you say the same for ddos protection? Isn't that the same as well?

[TechDebtDevin]

They arent doing anything. They are attempting to insert themselves into the middle of a marketplace (that doesnt exist and never will) where scrapers pay for IP. They think theyre going to profit off the bots, not protect your site. Dont fall for their scam.

[bitpush]

What do you mean they are trying to insert themselves? If I have a website that I host with cloudflare, I (as the rightful website owner) has inserted Cloudflare in between.

It isnt CF going around saying, that's a nice website you have there. I'm gonna put myself in between.

[TechDebtDevin]

They cant do anything other than bog down the internet. I havent found a single cf provided challenge I havent been able to get past in < half a day.

This is simply juat the first step in them implementing a marketplace and trying to get into LLM SEO. They dont care about your site or protecting it. They are gearing up to start making a cut in the Middle between scrapers and publishers. Why wouldnt I go DIRECTLY to the publisher and make a deal. So dumb I hate cf so much.

The only thing cloudflare knows how to do is MITM attacks.

[Marsymars]

So what would you suggest as an alternative if I have a site where I don’t want the content used for LLM training?

[fkyoureadthedoc]

Auth? Because whatever Cloudflare is doing isn't going to stop anyone serious about scraping data.

[Marsymars]

Let’s say I’m talking about content that I don’t want behind an auth wall. Is your position simply that all such sites should abandon any efforts to not have the content used for LLM training?

[fkyoureadthedoc]

CF will stop bots that respect your robots.txt, and try and stop ones that don't. If your concern is just that you don't want your content used to train an LLM, this will stop the honest companies.

If you are concerned about load on your site because the crawlers are hammering your site, the ones that respect robots.txt should be respecting your crawl delay too. CF will be able to block the dumb ones that ignore your robots.txt and hammer you with no real strategy.

But serious scrapers will have rotating residential IPs and be loading your site from real browsers, they'll take effort to appear as actual users. Sites like Ticket Master have an endless arms race against these. Some Chinese LLM company will get your data if it's public lol.

[Marsymars]

Sure, I understand all that, but you haven't really answered my question of what an alternative is, you've just laid out why CF is better than nothing. (Or even managing robots.txt manually.)

[Wilder7977]

Something like https://github.com/TecharoHQ/anubis?

It's not that different from CF, but you control it fully.

[mattl]

If you find a solution that’s not auth please let me know.

[DeusExMachina]

I would expect these features to be opt-in. Even though I agree with it, I would be pretty upset if they just turned it on automatically on my website.