[windsignaling]

As a website owner and VPN user I see both sides of this.

On one hand, I get the annoying "Verify" box every time I use ChatGPT (and now due its popularity, DeepSeek as well).

On the other hand, without Cloudflare I'd be seeing thousands of junk requests and hacking attempts everyday, people attempting credit card fraud, etc.

I honestly don't know what the solution is.

[rozap]

What is a "junk" request? Is it hammering an expensive endpoint 5000 times per second, or just somebody using your website in a way you don't like? I've also been on both sides of it (on-call at 3am getting dos'd is no fun), but I think the danger here is that we've gotten to a point where a new google can't realistically be created.

The thing is that these tools are generally used to further entrench power that monopolies, duopolies, and cartels already have. Example: I've built an app that compares grocery prices as you make a shopping list, and you would not believe the extent that grocers go to to make price comparison difficult. This thing doesn't make thousands or even hundreds of requests - maybe a few dozen over the course of a day. What I thought would be a quick little project has turned out to be wildly adversarial. But now spite driven development is a factor so I will press on.

It will always be a cat and mouse game, but we're at a point where the cat has a 46 billion dollar market cap and handles a huge portion of traffic on the internet.

[jeroenhd]

I've such bots on my server. Some Chinese Huawei bot as well as an American one.

They ignored robots.txt (claimed not to, but I blacklisted them there and they didn't stop) and started randomly generating image paths. At some point /img/123.png became /img/123.png?a=123 or whatever, and they just kept adding parameters and subpaths for no good reason. Nginx dutifully ignored the extra parameters and kept sending the same images files over and over again, wasting everyone's time and bandwidth.

I was able to block these bots by just blocking the entire IP range at the firewall level (for Huawei I had to block all of China Telecom and later a huge range owned by Tencent for similar reasons).

I have lost all faith in scrapers. I've written my own scrapers too, but almost all of the scrapers I've come across are nefarious. Some scour the internet searching for personal data to sell, some look for websites to send hack attempts at to brute force bug bounty programs, others are just scraping for more AI content. Until the scraping industry starts behaving, I can't feel bad for people blocking these things even if they hurt small search engines.

[MatthiasPortzel]

Why not just ignore the bots? I have a Linode VPS, cheapest tier, and I get 1TB of network transfer a month. The bots that you're concerned about use a tiny fraction of that (<1%). I'm not behind a CDN and I've never put effort into banning at the IP level or setting up fail2ban.

I get that there might be some feeling of righteous justice that comes from removing these entries from your Nginx logs, but it also seems like there's a lot of self-induced stress that comes from monitoring failed Nginx and ssh logs.

[x3haloed]

Honestly, it should just come down to rate limiting and what you’re willing to serve and to whom. If you’re a free information idealist like me, I’m OK with bots accessing public web-serving servers, but not OK with allowing them to consume all my bandwidth and compute cycles. Furthermore, I’m also not OK with legitimate users consuming all my resources. So I should employ strategies that prevent individual clients or groups of clients from endlessly submitting requests, whether the format of the requests make sense or are “junk.”

[makeitdouble]

Rate limiting doesn't help if the requests are split under hundreds of sessions. Especially if your account creation process was also bot friendly.

Fundamentally it's adversarial, so expecting a single simple concept to properly cover even half of the problematic requests is unrealistic.

[amatecha]

Rate limiting could help when an automated process is scanning arbitrary, generated URLs, inevitably generating a shitton of 404 errors -- something your rate limiting logic can easily check for (depending on server/proxy software of course). Normal users or even normal bots won't generate excessive 404's in a short time frame, so that's potentially a pretty simple metric by which apply a rate limit. Just an idea though, I've not done that myself...

[ku1ik]

I did that and it works great.

Specifically, I use fail2ban to count the 404s and ban the IP temporarily when certain threshold is exceeded in a given time frame. Every time I check fail2ban stats it has hundreds of IPs blocked.

[ghxst]

Rate limiting based on IP, blocking obvious datacenter ASNs and blocking identifiable JA3 fingerprints is quite simple and surprisingly effective in stopping most scrapers and can be done entirely server side, I wouldn't be surprised if this catches more than half of problematic requests to the average website. But I agree that if you have a website "worth" scraping there will probably be some individuals motivated enough to bypass those restrictions.

[dmantis]

> blocking obvious datacenter ASNs

You block all VPN users then, and currently many countries have some kind of censorship, please don't do that. I use a personal VPN for over 5 years and that's annoying.

I understand the other side and captcha/POW captchas/additional checks is okay. But give people a choice to be private/non-censorable.

Enabling/disabling a VPN each minute to access the non-censored local site which blocks datacenters IPs, then bringing it back again for the general surfing is a bit of a hell.

[DocTomoe]

Sounds like a problem easily solved with fail2ban. Which keeps legitimate folks in, and offenders out - and also unbans after a set amount of time, to avoid dynamic IPs screwing over legitimate users permanently.

[makeitdouble]

> somebody using your website in a way you don't like?

This usually includes people making a near-realtime updated perfect copy of your site and serving that copy for either scam or middle-manning transactions or straight fraud.

Having a clear category of "good bots" from either a verified or accepted companies would help for these cases. Cloudflare has such a system I think, but then a new search engine would have to go to each and every platform provider to make deals and that also sounds impossible.

[Terr_]

I'd settle for some kind of "proof of investment" in a bot-identity, so that I know blocking that identity is impactful, and it's not just one of a billion tiny throwaways.

In other words, knowing who someone is isn't strictly necessary, provided they have "skin the game" to encourage proper behavior.

[OptionOfT]

> and you would not believe the extent that grocers go to to make price comparison difficult. This thing doesn't make thousands or even hundreds of requests - maybe a few dozen over the course of a day.

It's gonna get even worse. Walmart & Kroger are implementing digital price tags, so whatever you see on the website will probably (purposefully?) be out of date by the time you get to the store.

Stores don't want you to compare.

[rozap]

Originally I was excited to see that kroger had an API, until just about the first thing that the ToS said was "you can't use this for price comparison".

And yea, I imagine dynamic pricing will make things even more complicated.

That being said, that's why this feature isn't built into the billion shopping list apps that are out there. Because it's a pain.

[unethical_ban]

Price comparison should be required by law. In fact, I think it would be interesting for a city to require its major grocers to feed pricing information to a public database.

[_blk]

So you put something in your cart and by the time you reach the cashier the price doubled? Sounds like someone is about to patent price locking when you add an item to your pysical shopping cart.

[ohcmon]

Actually, I think creating google alternative has never been as doable as it is today.

[to11mtm]

I'll give a fun example from the past.

I used to work at a company that did auto inspections. (e.x. if you turned a lease in, did a trade in on a used car, private party, etc.)

Because of that, we had a server that contained 'condition reports', as well as the images that went through those condition reports.

Mind you, sometimes condition reports had to be revised. Maybe a photo was bad, maybe the photos were in the wrong order, etc.

It was a perfect storm:

- The Image caching was all inmem

- If an image didn't exist, the server would error with a 500

- IIS was set up such that too many errors caused a recycle

- Some scraper was working off a dataset (that ironically was 'corrected' in an hour or so) but contained an image that did not exist.

- The scraper, instead of eventually 'moving on' would keep retrying the URL.

It was the only time that org had an 'anyone who thinks they can help solve please attend' meeting at the IT level.

> and you would not believe the extent that grocers go to to make price comparison difficult. This thing doesn't make thousands or even hundreds of requests - maybe a few dozen over the course of a day.

Very true. I'm reminded of Oren Eini's tale of building an app to compare grocery prices in Israel, where apparently mandated supermarket chains to publish prices [0]. On top of even the government mandate for data sharing appearing to hit the wrong over/under for formatting, There's the constant issue of 'incomparabilities'.

And it's weird, because it immediately triggered memories of how 20-ish years ago, one of the most accessible Best Buy's was across the street from a Circuit City, but good luck price matching because the stores all happened to sell barely different laptops/desktops (e.x. up the storage but use a lower grade CPU) so that nobody really had to price match.

[0] - https://ayende.com/blog/170978/the-business-process-of-compa...

[_factor]

Best Buy will also sell identical hardware with a slightly modified SKU and negligible changes to avoid comparison.

It’s difficult to compare when BB is the “only” company that sells a particular item.

[tempodox]

+1 for spite-driven development.

[gjsman-1000]

Simple: We need to acknowledge that the vision of a decentralized internet as it was implemented was a complete failure, is dying, and will probably never return.

Robots went out of control, whether malicious or the AI scrapers or the Clearview surveillance kind; users learned to not trust random websites; SEO spam ruined search, the only thing that made a decentralized internet navigable; nation state attacks became a common occurrence; people prefer a few websites that do everything (Facebook becoming an eBay competitor). Even if it were possible to set rules banning Clearview or AI training, no nation outside of your own will follow them; an issue which even becomes a national security problem (are you sure, Taiwan, that China hasn't profiled everyone on your social media platforms by now?)

There is no solution. The dream itself was not sustainable. The only solution is either a global moratorium of understanding which everyone respectfully follows (wishful thinking, never happening); or splinternetting into national internets with different rules and strong firewalls (which is a deal with the devil, and still admitting the vision failed).

[stevenAthompson]

I hate that you're right.

To make matters worse, I suspect that not even a splinternet can save it. It needs a new foundation, preferably one that wasn't largely designed before security was a thing.

Federation is probably a good start, but it should be federated well below the application layer.

[ToucanLoucan]

I mean, it wasn't even that security wasn't a thing: the earliest incarnations of the Internet were defense projects, and after that, connections between university networks. Abuse was nonexistent because you knew everyone on your given network. Bob up the hall wouldn't try to steal your credit card or whatever, because you'd call the police.

I think a decent idea is, we need to bring personal accountability back into the equation. That's how an open-trust network works, and we know that, because that's how society works. You don't "trust" that someone walking by your car won't take a shit in your open window: they could. But there are consequences for that. We need rock solid data security policies that apply to anyone who does business, hosts content, handles user data online, and people need to use their actual names, actual addresses, actual phone numbers, etc. etc. in order to interact with it. I get that there are many boons to be had with the anonymity the Internet offers, but it also enables all of the horseshit we all hate. A spammer can spam explicitly because their ISP doesn't care that they do, email servers don't have their actual information, and in the odd event they are caught and are penalized, it's fucking trivial to circumvent it. Buy a new AWS instance, run a script to setup your spam box, upload your database of potential victims, and boom, you're off.

A lot of tech is already drifting this way. What is HTTPS at it's core if not a way to verify you are visiting the real Chase.com? How many social networking sites now demand all kinds of information, up to and including a photo of your driver's license? Why are we basically forbidden now by good practice from opening links in texts and emails? Because too many people online are anonymous, can't be trusted, and are acting maliciously. Imagine how much BETTER the Internet would be if when you fucked around, you could be banned entirely? No more ban evasion, ever.

I get that this is a controversial opinion, but fundamentally, I don't think the Internet can function for much longer while being this free. It's too free, and we have too many opportunistic assholes in it for it to remain so.

[BrenBarn]

> It's too free, and we have too many opportunistic assholes in it for it to remain so.

There's some truth in this, but I think there is a lot of room for improving things as far as making life much more painful for opportunistic assholes in general.

[benatkin]

Me too.

Federation is indeed a good start, but DeFi helps spur adoption by having a broader scope.

[supportengineer]

A walled garden where each a real, vetted human being is responsible for each network device. It wouldn't scale but it could work locally.

[benatkin]

Luckily the decentralization community has always been decentralized. There are plenty of decentralized networks to support.

[Aeolun]

The great firewall, but in reverse.

[gjsman-1000]

What other choice do we have?

Countries, whether it be Ukraine or Taiwan, can't risk other countries harvesting their social media platforms for the mother of all purges. I never assume that anything that happened historically can never happen again - no Polish Jew would have survived the Nazis with this kind of information theft. Add AI into the mix, and wiping out any population is as easy as baking pie.

Countries are tired of bad behavior. Just ask my grandmother, who has had her designs stolen and mass produced from China. Not just companies - many free and open source companies cannot survive with such reckless competition. Can Prusa survive a world where China takes, but never gives? How many grandmothers does it take being scammed? How many educational systems containing data on minors need to be stolen? The MPAA and RIAA has been whining for years about the copyright problem, and while we laugh at them, never underestimate them. The list goes on and on.

Startups are tired of paying Cloudflare or AWS protection money, and trying to evade the endless sea of SEO spam. How can a startup compete with Google with so much trash and no recourse? Who can build a new web browser, and be widely accepted as being a friendly visitor? Who can build a new social media platform, without the experience and scale to know who is friend or foe?

Now we have AI, gasoline and soon to be dynamite on the fire. For the first time ever, a malicious country can VPN into the internet of a friendly nation, track down all critics on their social media, and destroy their lives in a real world attack (physical or virtual). We are only beginning to see this in Ukraine - are we delusional enough to believe that the world is past warfare? For the first time, anyone in the world could make nudes of women and share them online, from a location where they'll probably never be taken down. If a Russian company offered nudes as a service to American customers with cryptocurrency payments and a slick website that went viral, do you think tolerance is a winning political position?

[DocTomoe]

> Can Prusa survive a world where China takes, but never gives?

Can Prusa survive in a world where they only may operate in CzechNet, with the rest of the customer base being firewall off?

[gjsman-1000]

You’re assuming we couldn’t have mutually agreed upon interoperability treaties.

[rat87]

> no Polish Jew would have survived the Nazis with this kind of information theft.

I'm not sure this is a good is a good example. I believe a majority of Polish Jewish survivors were those who fled into parts of soviet union not occupied by nazis(some were sent to gulags but this was still much better chance to survive then those who stayed in Poland). Another large portion were in concentration camps and hadn't been killed yet. And I believe 60,000 or less are estimated to have hid in Poland through the war. It's unlikely many remained in their pre war identities and simply concealed their Jewishness and managed to survive.

[inetknght]

> On the other hand, without Cloudflare I'd be seeing thousands of junk requests and hacking attempts everyday, people attempting credit card fraud, etc.

Yup!

> I honestly don't know what the solution is.

Force law enforcement to enforce the laws.

Or else, block the countries that don't combat fraud. That means... China? Hey isn't there a "trade war" being "started"? It sure would be fortunate if China (and certain other fraud-friendly countries around Asia/Pacific) were blocked from the rest of the Internet until/unless they provide enforcement and/or compensation their fraudulent use of technology.

[marginalia_nu]

A lot of this traffic is bouncing all over the world before it reaches your server. Almost always via at least one botnet. Finding the source of the traffic is pretty hopeless.

[patrick451]

When the government actually cares, they're able to track these things down. But they don't except in high profile cases.

[marginalia_nu]

Well sometimes at least.

When the government really cares, it can put all its resources to solve any particular problem. Though obviously that comes at the cost of reassigning resources from other tasks. Sadly it's impossible to assign all resources to solve every problem all at once.

[jeroenhd]

A lot of the fake browser traffic I'm seeing is coming from American data centres. China plays a major part, but if we're going by bot traffic, America will end up on the ban list pretty quickly.

[inetknght]

America does have laws against this kind of thing.

So instead of banning America, report the IP addresses to their American hosts for spam and malicious intent. If the host refuses to do anything, report it to law enforcement. If law enforcement doesn't do anything... then you're proving my point.

[dmantis]

So you are saying that if 95% of world population, including Chinese, Russians, etc reports American bot farm to American police, somebody would really review that and go after Americans?

BTW, how they should report it, if they are a small business/physical person without lawyers? Does US police have some kind of online hotline to report US criminals for foreigners or smth?

[inetknght]

It's almost as if there should be an international body of laws which covers fraud...

[dmantis]

That's not feasible for bots, crawling, IP laws, etc.

Strict fraud could be handled, but everything above is really different per jurisdiction by obvious reasons. There is nothing clearly good or bad in bots, or e.g. pirates, it depends on particular cultural perception. And if one nation thinks that the action is not a crime, it doesn't make sense to them to prosecute such actions for foreign requests.

[portaouflop]

How are you gonna force law enforcement to enforce the laws?

[inetknght]

Thanks for finding my point!

[jacobr1]

Slightly more complicated because a ton of the abuse comes from IPs located western countries, explicitly to evade fraud and abuse detection. Now you can go after the western owners of those systems (and all the big ones do have have large abuse teams to handle reports) but enforcement has a much higher latency. To be effective you would need a much more aggressive system. Stronger KYC. Changes in laws to allow for less due-process and more "guilty by default" type systems that you then need to prove innocence to rebut.

[warkdarrior]

And that assumes that the Western owners of those systems have any reason to listen to you, the one raising the complaint. How would they check that you are not lying?

[RIMR]

A wild take only possible if you don't understand how the Internet works.

[inetknght]

A wild opinion only valid if you have a defeatist attitude.

[EVa5I7bHFq9mnYK]

Credit card fraud exists because credit card companies can't (or won't) implement elementary security measures. There should be a requirement to confirm every online payment, but many sites today require just a cc number+date+code+zip, with no additional confirmation, can't call it other than complicity in the crime.

[il-b]

Lost sales due to 2fa are greater than losses due to refunds

[xrisk]

Why would 2FA cause lose sales? One would imagine it’s because people are being auto charged for shit they don’t want but haven’t noticed or forgot to cancel.

[EVa5I7bHFq9mnYK]

Because it's more work? Also 2fa often fails for the rightful card owner. And Cloudflare overzealous "security" is one of the reasons for failure.

[simplyinfinity]

in europe 2fa is mandatory for all (or almost all) online purchases, especially first time purchase from a merchant when your card hasn't been authorized. Sites using stripes' link get away with no 2fa most of the time, but not all the time. Make it mandatory on visa/mastercards level, and you won't loose much sales, as all transactions would require it and people will have to 2fa everywhere.

[EVa5I7bHFq9mnYK]

An hour ago paid to Contabo cloud service provider, headquartered in Munich. No 2fa.

[BrenBarn]

Yeah, and this is actually a huge pain for visitors. I was in Europe a couple months ago and couldn't buy stuff like train tickets online. Why? Because everything wants to verify with a text, and I couldn't do that because I had gotten a European SIM card because my US plan doesn't do international roaming.

There are several colliding problems there (cheap cell phone plan, 2fa being via text, online purchases requiring 2fa) but it still illustrates to me the pain of doing simple stuff in the modern tech space. I wish the powers that be would work harder on solutions that don't require extra work from the people doing small, normal stuff. It would be better to have a lot more fraud occur but a lot more of the perpetrators pursued and caught. A lot of anti-fraud measures seem to be largely about passing the buck to someone else instead of actually eliminating the humans who are driving the fraud.

[crazygringo]

Because it just doesn't work with shocking frequency.

Maybe 10% of the time I make a purchase online, it shows me a screen where it says it's waiting for my bank to verify, I'll have to input a code or accept a notification or something.

A solid half the time it fails. Either the site decides the transaction was rejected before I even get a chance to respond (within seconds), or I just don't get any notification or code or anything, or I do authorize it and it still gets rejected.

[xrisk]

idk here in India, we have 2FA for everything. I would say it very rarely fails, speaking from personal experience.

[crazygringo]

I think a lot of other countries have it much more standardized. Or it's just more common so the bugs get fixed.

But in the US there are so many credit card providers, each one seems to do it differently, and the UX flows just break. And it seems difficult for a site to even test, and how will you even figure out if it's the provider or network or merchant or notification that's failing?

[BytesAndGears]

Something like iDeal, which is a payment processing system in the Netherlands.

It works so well and is very secure. You get to the checkout page on a website, click a link. If you’re on your phone, it hotlinks to open your banking app. If you’re on desktop, it shows a QR code which does the same.

When your bank app opens, it says “would you like to make this €28 payment to Business X?” And you click either yes or no on the app. You never even need to enter a card in the website!

You can also send money to other people instantly the same way, so it’s perfect for something like buying a used item from someone else.

Plus the whole IBAN system which makes it all possible!

[carlosjobim]

What kind of fraud protection does iDeal have for customers?

[BytesAndGears]

I’m not actually sure since I never had issues, but I’ve heard it’s not much since they’re basically just an API for transferring money between banks. Each bank app still needs to integrate with the network separately. [1]

I guess you get some security since each party that you transfer to must have their identity verified with a bank, so you could always get the police involved fairly easily

The iDeal website page on security [2] is in Dutch, but it translates to roughly:

> Before you make a purchase, make sure that the webshop or business is a reliable party. For example, you can read experiences of other consumers about webshops on comparison sites. Or you can use a Google search to check what is said (in reviews) about a webshop on the internet. Also check the overview of the police with known rogue trading parties and the page check seller data. Before making a purchase, always use the following rule of thumb: if something is too good to be true, don't do it.

[1] https://en.m.wikipedia.org/wiki/IDEAL

[2] https://www.ideal.nl/veiligheid

[carlosjobim]

Thank you for the explanation. Then I think credit/Debit cards are a better option for the customer, considering they have fraud protection.

[kobalsky]

> people attempting credit card fraud

this is wrong.

if someone can use your site they can use stolen cards, and bots doing this will not be stopped by them.

cloudflare only raises the cost of doing it, it may make scrapping a million of product pages unprofitable but that doesn't apply to cc fraud yet.

[hecanjog]

They might be talking about people who are trying to automate the testing hundreds of stolen credit cards with small purchases to see if they are still working. This is basically why we ended up using cloudflare at work.

[bragr]

>that doesn't apply to cc fraud yet

It stops "card testing" where someone has bought or stolen a large number of cards and need verify which are still good. The usual technique is to cycle through all the cards on a smaller site selling something cheap (a $3 ebook for example). The problem is that the high volume of fraud in a short time span will often get the merchant account or payment gateway account shut down, cutting off legitimate sales.

As a consumer, you should also be suspicious of a mysterious low value charge on your card because it could be the prelude to much larger charges.

[Aachen]

Someone who steals money from thousands of individuals for a living won't hesitate to use a botnet either. Cloudflare isn't a payment provider (*shudders* yet), they can't verify transactions, they can only guess at who's "honest". I'm at the losing end of this guess so often as someone who frequently visits friends and family in the neighbouring country they come from, and someone who doesn't have tracking cookies anymore that were set only a few minutes ago, who uses a "non-standard" browser (Mozilla's Firefox), I don't feel like Cloudflare does a very good job at detecting when I'm trying to honestly use the site. At the same time, doing security testing as my job: the customer having Cloudflare enabled usually doesn't matter for us being able to reach and exploit vulnerable pages, it just decides to block you randomly the same way that it does in private time when I'm not trying to break anything. It doesn't properly do the job and it blocks legitimate people based on a gut feeling, and you have no recourse, you can suck it up. Whatcha gonna do, take Cloudflare to court for blocking your access to your bank? Under what law is that illegal? There is nothing you can do; your bank's customer support isn't going to disable Cloudflare for you.

Anyway, no, this guessing game isn't the solution to stolen bank details, the solution is for the payment provider to authenticate the account holder beyond merely entering a public number, especially if they suddenly see a flood of transactions from this one merchant as you describe. They can decide to ask for a second factor: send the person an SMS/email, ask to generate an authenticator code, whatever it is they've got on file beyond your card/account number. Anything else is just guesswork

[doctor_radium]

> Whatcha gonna do, take Cloudflare to court for blocking your access to your bank? Under what law is that illegal?

In the USA, I think it would be worth trying to sue Cloudflare for either "free speech" or "public nuisance" violations. Gonna reach out to the ACLU and EFF in the coming days.

[lmz]

It depends what they're selling. If they're selling something people want - the only answer is enforcing things like 3DS. If they are e.g. a charity receiving donations via card - they may still use it for card testing. Making card testing unprofitable is the point.

[markisus]

If I were hosting a web page, I would want it to be able to reach as many people as possible. So in choosing between CDNs, I would choose the one that provides greater browser compatibility, all other things equal. So in principle, the incentives are there for Cloudflare to fix the issue. But the size of the incentive may be the problem. Not too many customers are complaining about these non-mainstream browsers.

[porty]

In that case you can turn off / not turn on the WAF feature(s) of Cloudflare - it's optional and configured by the webmaster.

[doctor_radium]

On one hand, I'm okay with that. If Cloudflare or some other self-appointed Internet cop blocks me from a site, I just go somewhere else, and I hope the site goes out of business as a result...which happens to businesses everyday for a variety of reasons. But given Cloudflare's sheer size, having so many businesses crank the shields to maximum actually affects using the web, and that's where I draw the line.

[Aachen]

> If I were hosting a web page, I would want it to be able to reach as many people as possible. So in choosing between CDNs

I host many webpages and this is exactly it. Anyone is welcome to use the websites I host. There is no CDN, your TLS session terminates at the endpoint (end to end encryption). May be a bit slower for the pages having static assets if you're coming from outside of Europe, but the pages are light anyway (no 2 MB JavaScript blobs)

[lynndotpy]

> On the other hand, without Cloudflare I'd be seeing thousands of junk requests and hacking attempts everyday, people attempting credit card fraud, etc. > > I honestly don't know what the solution is.

The solution is good security-- Cloudflare only cuts down on the noise. I'm looking at junk requests and hacking attempts flow through to my sites as we speak.

[lynndotpy]

Whoops-- this was a draft I didn't intend to post in this state. I must have fatfingered the "reply" button somehow. Alas, too late to edit or delete now.

Cloudflare cuts down on the noise, but also helps does the work of preventing scrapers, people who re-sell your site wholesale, and cutting down on the noise also means cutting down on the cost of network requests.

It also can help where security is lax. You should have measures against credential stuffing, but if you don't, Cloudflare might prevent (some) of your users from being hacked. Which isn't good enough, but is better than no mitigation at all.

I don't use Cloudflare personally, but I won't dismiss it wholesale. I understand why people use it.

[carlosjobim]

>Cloudflare only cuts down on the noise.

That sounds like the solution, that sounds like good security.

[boomboomsubban]

>On one hand, I get the annoying "Verify" box every time I use ChatGPT (and now due its popularity, DeepSeek as well).

Though annoying, it's tolerable. It seemed like a fair solution. Blocking doesn't.

[chaoskitty]

Simple: Don't look at the logs.

Bots are a fact of life. Secure your site properly, follow good practices, set up notifications for important things, log stuff, but don't look at the logs unless you have a reason to look at the logs.

Having run web servers forever, this is simply normal. What's not normal is blindly trusting a megacorporation to make my logs quiet. What're they doing? Who are they blocking? What guidelines do they use? Nobody, except them, knows.

It's why I self-host email. Sure, you might feel safe because most people use Gmail or Outlook, and therefore if there are problems, you can point the finger at them, but what if you want to discuss spam? Or have technical discussions about Trojans and viruses? Or you need to be 100% absolutely certain that email related to specific events is delivered, with no exceptions? You can't do that with Gmail / Outlook, because they have filters that you can't see and you can't control.

[buyucu]

My VPN/Fileserver VPS is not behind Cloudflare, and I haven't had any trouble for years. Only the SSH port is accessible from outside (which is probably not even necessary), with password login disabled. I use fail2ban and a few other extra layers of security.

[buyucu]

Credit cards are an ancient insecure technology that needs to go away. There are systems in Europe like iDEAL that are much more 21st century appropriate.

[grayhatter]

> I honestly don't know what the solution is.

well, for starters, if you're using cloudflare to block otherwise benign traffic, just because you're worried about some made... up....

> On the other hand, without Cloudflare I'd be seeing thousands of junk requests and hacking attempts everyday, people attempting credit card fraud, etc.

well damn, if you're using it because otherwise you'd be exposing your users to active credit card fraud... I guess the original suggestion to only ban traffic once you find it to be abusive, and then only by subnet, doesn't really apply for you.

I wanna suggest using this as an excuse to learn how not to be a twat (the direction cf is moving towards more and more), where for most sites 20% of the work will get you 80% of the results... but dealing with cc fraud, you're adversaries are already on the more advanced side, and that becomes a lot harder to prevent... rather than catch and stop after the fact.

Balancing the pervasive fear mongering with sensible rules is hard. Not because it's actually hard, but because that's the point of the FUD. To create the perception of a problem where there isn't one. With a few exceptions, a WAF doesn't provide meaningful benefits. It only serves to lower the number of log entries, it rarely ever reduces the actual risk.

[ludjer]

I used to work one of the top 1000 visited websites, and we have massive bot issues where 60% of our traffic was bots and had to implement solutions similar to cloudflare to reduce the bots. Also, with the raise of ai, it's become even more important since a lot of ai data scraping companies do not respect robots.

[jillyboel]

accept reality and design your api so it's not a problem