[tekacs]

> For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default.

It's kind of exhausting seeing Cloudflare playing both sides of the arms race.

I just can't imagine bringing myself to use their technology to build agents and build AI products when they're also doing things like this.

> This also lines up the incentive model we want to foster. Losing trusted status across the more than 20% of web domains that sit behind Cloudflare is a deterrent with teeth. Trust becomes something you can carry with you, and something you can lose.

And even more so, LLM language aside, fun and fascinating to see them flagrantly calling out their position here as if it's a positive.

[usef-]

How are they playing both sides? I thought their scraping products were also about having it behave and not take down systems

[inigyou]

Their scraping products are mostly about paying them money to not be blocked.

[siva7]

So Google has to pay Cloudflare 10B$ to get Googlebot moved to their default Allowlist otherwise Google would loose access to a significant portion of the internet? Genius move.

[inigyou]

Yes literally this. Every large corporation gatekeeper does this sort of thing, it's why we have to militantly push for decentralisation by avoiding these corporations. It's not morally different from Google forcing your site to be AI-scrapeable or be kicked off Google Search. It's why Apple won't let you publish an app without paying them 30% of your total revenue. It's why Microsoft threatened to ban Steam (they later decided not to) so they could get all those games onto the Microsoft Store, and in response Valve ported gaming to Linux.

[schainks]

Wait until you find out what Google is paying Reddit…

[tekacs]

I mean that they're telling developers that they should use Cloudflare's platform to build agents, the kind of agents that would go across the web and act on behalf of users... but then they're also the ones blocking those requests.

This always engenders a solid amount of distaste from me, because much like Google and Chrome, it creates the incentive for you to treat yourself better than others. Especially coupled with the trust stuff. Of course, Cloudflare is always going to trust their own platform.

[samrus]

There are ways to use those agents ethically. Cloudflare isnt forcing you to be a dick with your agents. Infact the blocking side encourages you to use them ethically. I dont see the problem.

[stingraycharles]

I thought the issue people have with AI scrapers is the ones that DDoS sites to scrape everything for training purposes, rather than the ones that interactively query specific content to support an active conversation?

[tekacs]

If you read the article, you'll notice that they're explicitly making sure to block the ones that interactively query specific content too.

[PunchyHamster]

First one is problem because it brings no traffic back and occasionally DDoSes the site

Second one is problem because it DDoSes the site. "Just" queries for hundred thousand people (if you happened to be good source for that bit of knowledge) that don't bring actual people to your site is also a problem

[inigyou]

First one is not occasional, it is the anonymous global adversary probably ddosing your site right now. It never relents.

[nirui]

> not take down systems

"Block on pages with ads" is probably about preventing the AI crawlers from clicking on the ads which maybe considered cheating by the ad company.

If you want to prevent "bot attacks", maybe the "Block" option will do the trick.

But of course, to do all that you need to put some trust on Cloudflare, because they're the one identifying the bots from normal users.

For me, as someone who's hosting a Gitea instance behind Cloudflare, I have a Configuration Rule set that says: if the client is trying to access a URL that is beyond certain length limit, then trigger "Browser Integrity Check" and "I’m Under Attack", a.k.a stricter security checks.

The match expression of the rule looked something like this:

    (
      len(http.request.uri) > !!!!!SET LENGTH LIMIT!!!!! and
      not lower(http.request.uri.path) contains ".git/" and
      not lower(http.request.uri.path) contains "api/"
    )

(The `!!!!!SET LENGTH LIMIT!!!!!` is an integer of the length limit you wanted to set)

This rule alone basically blocked all abusive bot traffic to almost zero for my site (https://i.imgur.com/LaOjjvV.png, see the traffic drop around 10 clock and Cloudflare mitigation kicks in).

But of course, you need to figure out your own rules based on the characteristic of the website. Also, you can be more creative: for example, my actual rule is more complex than that, it also checks to see if a cookie is not set, and only triggers when all condition are met:

    (
      len(http.request.uri) > !!!!!SET LENGTH LIMIT!!!!! and
      not lower(http.request.uri.path) contains ".git/" and
      not lower(http.request.uri.path) contains "api/" and
      not http.cookie wildcard "*!!!!!COOKIE NAME!!!!!=!!!!!COOKIE VALUE!!!!!*"
    )

then, as part two of that rule, I have a Response Header Transform Rules that says:

    (
      len(http.request.uri) <= !!!!!SET LENGTH LIMIT!!!!! and
      not http.cookie contains "!!!!!COOKIE NAME!!!!!=!!!!!COOKIE VALUE!!!!!"
    )

and if this Response Header Transform Rules is triggered, it sets the cookie `!!!!!COOKIE NAME!!!!!=!!!!!COOKIE VALUE!!!!!`.

(Note: `!!!!!COOKIE NAME!!!!!` and `!!!!!COOKIE VALUE!!!!!` are the variables you need to customize)

When you put the two rules together, it forces clients to access "shallow" (short URL) pages first as an user would normally do, before they can access "deeper" (long URL) content hosted on the site without triggering more strict security checks. If that makes sense.

Also, don't forget the cookie basically also dug a hole in the security setting. So it's really a balance between avoid annoying the user and protect your site. You need to be smart and be flexible about it, otherwise your users will just leave.

[inigyou]

> If you want to prevent "bot attacks", maybe the "Block" option will do the trick.

This morning, for the first time, I clicked on a link posted on HN and was told in no uncertain terms that I am a bot and will not be allowed to view this page. By Cloudflare.

Your strategy is fine and similar to one of the checks in go-away. The point is that the unknown global DDoS adversary is using a very simple scraper, which does not load images or CSS or scripts, does not set cookies, etc

[nirui]

Yes, it only work if the scraper, or rather scrapers cannot collaborate in a smart way (for example, doing Cookie sharing among themselves). That's limited by how Cloudflare page rule work, it don't really support anything that is too "dynamic", for example you can't concat/mix a Cookie with a Request date and then compare it, you can only check if there's a Cookie or Request date match a static value.

If you want something more powerful, there's also Cloudflare workers, which you can program to do whatever you want. But that one may cost you to run.

Also in my case, the scraper IPs are all over the globe, and almost each access is from a different one, so simple IP blocking don't work unless I block Vietnam, Brazil, Bangladesh, Argentina and the US etc (https://i.imgur.com/eWI5meM.png). But then, the scrapers might just go to another country next month.

Since I deployed those rules, the access from scrapers has almost stopped, while normal search engine crawls are almost unaffected. So it do still work for me, even if it's very rudimentary. But hey, security though obscurity it alone is not enough, but it sometimes do delay the attacks.

[colechristensen]

I see Cloudflare as trying to forge the appropriate path ahead. Neither allowing the free-for-all nor trying to block everything isn't playing both sides, it's the path straight down the middle. Providing tools for producers and consumers to do things with permission and compensation.