Hacker News new | ask | show | jobs
by idoubtit 15 days ago
That's interesting. I haven't used fail2ban for a long time, but reaction is worth evaluating. Unfortunately, that post does not describe their full configuration. Maybe it's on purpose, so that attackers can't adjust to fit.

My experience is that modern web scraping had no obvious pattern, since it is proxied through many IPs. The last time a server was failing to handle the pressure, we decided to temporarily ban IPs from some Asian regions. How does the FSF decide to ban an IP?

Why do they use iptables + ipset instead of nftables? Is there a technical reason or is it just legacy? AFAIK, Nftables is more performant, and IMO simpler. And it has native sets, see https://wiki.nftables.org/wiki-nftables/index.php/Sets
2 comments
itintheory 14 days ago
> The last time a server was failing to handle the pressure, we decided to temporarily ban IPs from some Asian regions.

This is something we've been forced to do at work, a LOT. Some weeks it's Huawei Cloud, Tencent, and Alibaba. Other weeks it's all China Telecom. We're using Anubis where possible, but a lot of it is just whack-a-mole with residential proxies. I looked at Datadome and HUMAN, but they would be hundreds of thousands a year at our traffic scale, and I suspect may also have false positives. We abandoned CrowdSec for that reason as well.

I'd love to find a decent k8s native solution to this problem.

link
arbol 7 days ago
Prosopo could cut out a lot of the residential proxy nonsense for you. We integrate with lambda@Edge / cloudfront workers / server side and perform analytics to detect residential proxy networks - at far less cost than DD or HUMAN.
link
Hackbraten 14 days ago
> I looked at Datadome and HUMAN, but they would be hundreds of thousands a year at our traffic scale, and I suspect may also have false positives.

I can confirm. DataDome has been making my life a living hell. It thinks my phone is a bot, so I can no longer use PayPal and other quasi-monopoly services while on the go. DataDome will reliably block me on the first request.

Fuck DataDome.

link
thomzane 14 days ago
iptables has been mostly a wrapper for nftables for some time now. The choice of iptables + ipset with reaction is the difference in their configuration. Compare restart performance between the ipset and nftables example configurations with lists of greater than 1 million IPs.
link