[marticode]

As a user I do care, because I waste so much time on Cloudflare's "prove you are human" blocking-page (why do I have to prove it over and over again?), and frequently run on websites blocking me entirely based on some bad IP-blacklist used along with Cloudflare.

[tempest_]

Unfortunately the internet sucks in 2025.

If you have a site with valuable content the LLM crawlers hound you to no end. CF is basically a protection racket at this point for many sites. It doesnt even stop the more determined ones but it keeps some away.

[seniorThrowaway]

Yep for anyone unaware of how awful things truly are, look up what a "residential proxy" is. Back in my day we called that a botnet.

[nananana9]

Oh, they're still botnets. We just look the other way because they're useful.

And they're pretty tame as far as computer fraud goes - if my device gets compromised I'd much rather deal with it being used for fake YouTube views than ransomware or a banking trojan.

[deadbabe]

You can make a little bit of cash on the side letting companies use your bandwidth a bit for proxying. You won’t even notice. $50/month. Times are tough!

[jamwil]

Of course the risk here being whatever nefarious or illegal shit is flowing through your pipes, which you consented to and even received consideration for.

[deadbabe]

No worries it’s encrypted traffic

[collinmanderson]

> If you have a site with valuable content the LLM crawlers hound you to no end.

The site doesn't even need to have valuable content. Any content at all.

[hollerith]

CF would be a protection racket only if CF is the cause of the problem CF is charging money to solve.

[j2kun]

And yet half the HN front page every day is promoting LLM stuff.

"The internet sucks", yes, but we're doing it to ourselves.

[roflyear]

Unfortunately the problem isn't just "the internet sucks" it's "the internet sucks, and everyone uses it" - meaning people are not doing stuff offline, and a lot of our lives require us to be online.

[worik]

The Internet is huming along beautifully

It is the Web that is being degraded

[kadushka]

Would you rather not have LLMs?

[foobarchu]

Absolutely. They have dramatically worsened the world, with little to no net positive impact. Nearly every (if not all) positive impacts have an associated negative that that dwarfs it.

LLMs aren't going anywhere, but the world would be a better place if they hadn't been developed. Even if they had more positive impacts, those would not outweigh the massive environmental degradation they are causing or the massive disincentive they created against researching other, more useful forms of AI.

[kazen44]

LLM's to me sound like a "boiling the ocean" kind of approach to solving a problem.

[j2kun]

IMO LLMs have been a net negative on society, including my life. But I'm merely pointing out the stark contrast on this website, and that fact that we can choose to live differently.

[kadushka]

Are you anti-AI in general, or are you unhappy about the current LLMs?

[j2kun]

I am not anti-AI, nor unhappy about how any current LLM works. I'm unhappy about how AI is used and abused to collective detriment. LLM scraper spam leading to increased centralization and wider impacting failures is just one example.

[ToucanLoucan]

Yes.

A solid secondary option is making LLM scraping for training opt-in, and/or compensating sites that were/are scraped for training data. Hell, maybe then you could not knock websites over incentivizing them to use Cloudflare in the first place.

But that means LLM researchers have to respect other people's IP which hasn't been high on their todo lists as yet.

bUt ThAT dOeSn'T sCaLe - not my fuckin problem chief. If you as an LLM developer are finding your IP banned or you as a web user are sick of doing "prove you're human" challenges, it isn't the website's fault. They're trying to control costs being arbitrarily put onto them by a disinterested 3rd party who feels entitled to their content, which it costs them money to deliver. Blame the asshole scraping sites left and right.

Edit: and you wouldn't even need to go THAT far. I scrape a whole bunch of sites for some tools I built and a homemade news aggregator. My IP has never been flagged because I keep the number of requests down wherever possible, and rate-limit them so it's more in line with human like browsing. Like so much of this could be solved with basic fucking courtesy.

[lenerdenator]

Not to speak for the other poster, but... That's not a good-faith question.

Most of the problems on the internet in 2025 aren't because of one particular technology. They're because the modern web was based on gentleman's agreements and handshakes, and since those things have now gotten in the way of exponential profit increases on behalf of a few Stanford dropouts, they're being ignored writ large.

CF being down wouldn't be nearly as big of a deal if their service wasn't one of the main ways to protect against LLM crawlers that blatantly ignore robots.txt and other long-established means to control automated extraction of web content. But, well, it is one of the main ways.

Would it be one of the main ways to protect against LLM web scraping if we investigated one of the LLM startups for what is arguably a violation of the Computer Fraud and Abuse Act, arrested their C-suite, and sent each member to a medium-security federal prison (I don't know, maybe Leavenworth?) for multiple years after a fair trial?

Probably not.

[chasing0entropy]

I'm Sure there will be an investigation... By the SEC when the bubble pops and takes the S&P with it. No prison though, probably jobs at the next ponzi scheme

[j2kun]

Well said.

[captainkrtek]

hard yes, all of the technical discussion aside, the constant advertising deluge of every company touting AI is mind numbing.

[seanw444]

It's helped me learn some things quicker, but I definitely prefer the old days.

[salawat]

Can I raise that to no LLMs or SEO?

[worik]

Yes

LLMs have become a crucial compendium of knowledge, that had become hidden behind SEO

[davidhaymond]

Absolutely. And while we're at it, let's do away with social media.

[BrenBarn]

Good lord yes. No question.

[stalfosknight]

Yes

[inferiorhuman]

Yes

[nhhvhy]

Yes.

[LtWorf]

Yes.

[therein]

Yes, they are terrible and more a negative force than a positive one in every way imaginable. I would take no LLMs all day every day.

[pixl97]

I'd also take no war, no murder, and no disease, but that's not the world we live in.

[woooooo]

I just realized, why don't they have some "definitely human" third party cookie that caches your humanness for 24h or so? I'm sure there's a reason, I've heard third party cookies were less respected now, but can someone chime in on why this doesn't work and save a ton of compute?

[acureau]

Because people will solve the challenge once, and then use the cookie in automation tools. It already happens with shorter expiration cookies.

[woooooo]

Thanks, I'm now shaking my head at my naivete :)

[basilikum]

https://developers.cloudflare.com/waf/tools/privacy-pass/

[arbol]

Are you really posting this today?

[octoberfranklin]

Yes, there are several, and the good one (linked below) lets you use the "humanness" token across different websites without them being able to use it as a tracking signal / supercookie. It's very clever.

https://github.com/ietf-wg-privacypass/base-drafts

https://privacypass.github.io/

[lotsofpulp]

I assume that will be for Apple (and eventually Alphabet) to implement via digital IDs linked to real world IDs.

https://www.apple.com/newsroom/2025/11/apple-introduces-digi...

[philipwhiuk]

Don't worry, Sam Altman is selling the protection too -- https://en.wikipedia.org/wiki/World_(blockchain)

[crazygringo]

But that's not a problem caused by Cloudflare.

That's a problem caused by bots and spammers and DDoSers, that Cloudflare is trying to alleviate.

And you generally don't have to prove it over and over again unless there's a high-risk signal associated with you, like you're using a VPN or have cookies disabled, etc. Which are great for protecting your privacy, but then obviously privacy means you do have to keep demonstrating you're not a bot.

[BarryMilo]

You might say the problem CloudFlare is causing is lesser than the ones it's solving, but you can't say they're not causing a new, separate problem.

That they're trying counts for brownie points, it's not an excuse to be satisfied with something that still bothers a lot of people. Do better, CloudFlare.

[crazygringo]

Do better, how?

If you have any ideas on how to protect against bad actors in a way that is just as effective but easier for users, please share it.

Because as far as I can tell, this isn't a question of effort. It's a question of fundamental technological limitations.

[foresto]

"We have decided to endlessly punish you for using what few tools you have to avoid being exploited online, because it makes our multi-billion dollar business easier. Sucks to be you."

[crazygringo]

Small non-profits and personal blogs encounter the same problem. It has nothing to do with being a multi-billion dollar corp.

If you have a better technological solution, we'd all love to know it. Because right now, site owners are using the best tools available.

Criticizing when there's no other solution isn't very useful, is it?

[foresto]

> It has nothing to do with being a multi-billion dollar corp.

Cloudflare is the multi-billion dollar corporation. It has everything to do with that, because they are the primary cause, and their resources and position make them by far the best equipped to solve it.

> Criticizing when there's no other solution isn't very useful, is it?

Of course it is. Without criticism, the growing problem goes unacknowledged and allowed to persist. It should instead be continually called out until it is prioritized, and some of those billions should be spent on researching a solution. (Similarly, a company found to be dumping waste into a river should be held responsible for cleaning up the mess they created. Even if that turns out to be expensive or difficult.)

Expecting a single affected person to solve it for the big corp that caused it is unrealistic. And blaming the victims because they use VPNs or disable cookies is... unhelpful.

[crazygringo]

That's an absurd position to take.

CloudFlare is protecting sites from DDoS attacks and out-of-control bots. They're not the ones causing them. If CloudFlare wasn't asking you to prove you're human, many times the site would be down entirely because it couldn't keep up. Or the site would simply shut down because it couldn't afford it.

And this isn't a question of spending some fraction of billions on researching a solution. There fundamentally isn't one, if you understand how the internet works. This is a problem a lot of people would like to solve better, believe me.

So, yes, criticizing Cloudflare here is as useful as criticizing it for not having faster-than-light communication. There's nothing else it can do. It's not "blaming the victims".

I'm going to assume you simply don't have the technical understanding of how the internet works. Because the position you're taking is simply absurd and nonsensical, and there's no way you would write what you're writing otherwise.

[edm0nd]

Congrats, you now know what it's like to be a daily Tor user trying to hit normie sites from exit node IPs xD

[replwoacause]

Why would anyone be a daily Tor user and trying to hit clear-net sites on top of that? This sounds like a bizarre usecase.

[pinko]

Privacy through uniformity, operational security by routine, herd immunity for privacy, traffic normalization, "anonymity set expansion", "nothing to hide" paradox, etc.

I.e., if you use Tor for "normie sites", then the fact that someone can be seen using Tor is no longer a reliable proxy for detecting them trying to see/do something confidential and it becomes harder to identify & target journalists, etc. just because they're using Tor.

[replwoacause]

Huh never thought about that. I wonder how many people do that? Seems like a public service.

[milderworkacc]

It certainly feels like one at times!

[edm0nd]

Tor Browser has ~1M daily users. Tons of people use it for hitting sites that may be blocked in their country or they want to have some privacy like view pregnancy or health related articles and etc.

[dooglius]

In addition to the reasons in sibling comment, this also acts as a filter for low-quality ad-based sites; same reason I close just about any website that gives me a popup about a ToS agreement.

[jakub_g]

I hate it as much (and the challenge time seems to be getting longer, 10s lately for me, what the hell?)

But we can all say thank you to all the AI crawlers who hammer websites with impossible traffic.

[pixl97]

I mean, it was a problem before AI crawlers with just bots and attacks in general.

[olyjohn]

It wasn't nearly as bad.