[swiftcoder]

Making the obviously-abusive bot prohibitively expensive is one way to go, if you control the terminating server.

gzip bomb is good if the bot happens to be vulnerable, but even just slowing down their connection rate is often sufficient - waiting just 10 seconds before responding with your 404 is going to consume ~7,000 ports on their box, which should be enough to crash most linux processes (nginx + mod-http-echo is a really easy way to set this up)

[gildas]

Great idea, some people have already implemented it for the same type of need, it would seem (see the list of user agents in the source code). Implementation seems simple.

https://github.com/0x48piraj/gz-bomb/blob/master/gz-bomb-ser...

[kijin]

Be careful using this if you're behind cloudflare. You might inadvertently bomb your closest ally in the battle.

[mkj]

AWS customers have to pay for outbound traffic. Is there a way to get them to send you (or cloudflare) huge volumes of traffic?

[horseradish7k]

yeah, could use a free worker

[compootr]

free workers only get 100k reqs per day or something

[_pdp_]

A KB zip file can expand to giga / petabytes through recursive nesting - though it depends on their implementation.

[sim7c00]

thats traffic in the other direction

[swiftcoder]

The main joy of a zip bomb is that it doesn't consume much bandwidth - the transferred compressed file is relatively small, and it only becomes huge when the client tries to decompress it in memory afterwards

[crazygringo]

It's still going in the wrong direction.

[dns_snek]

It doesn't matter either way. OP was thinking about ways to consume someone's bandwidth. A zip bomb doesn't consume bandwidth, it consumes computing resources of its recipient when they try to unpack it.

[CWuestefeld]

We've been a similar situation. One thing we considered doing is to give them bad data.

It was pretty clear in our case that they were scraping our site to get our pricing data. Our master catalog had several million SKUs, priced dynamically based on availability, customer contracts, and other factors. And we tried to add some value to the product pages, with relevant recommendations for cross-sells, alternate choices, etc. This was pretty compute-intensive, and the volume of the scraping could amount to a DoS at times. Like, they could bury us in bursts of requests so quickly that our infrastructure couldn't spin up new virtual servers, and once we were buried, it was difficult to dig back out from under the load. We learned a lot during this period, including some very counterintuitive stuff about how some approaches to queuing and prioritizing that appeared sounded great on paper, actually could have unintended effects that made such situations worse.

One strategy we talked about was that, rather than blocking the bad guys, we'd tag the incoming traffic. We couldn't do this perfect accuracy, but the inaccuracy was such that we could at least ensure that it wasn't affecting real customers (because we could always know when it was a real, logged-in user). We realized that we could at least cache the data in the borderline cases so we wouldn't have to recalculate (it was a particularly stupid bot that was attacking us, re-requesting the same stuff many times over); from that it was a small step to see that we could at the same time add a random fudge factor into any numbers, hoping to get to a state where the data did our attacker more harm than good.

We wound up doing what the OP is now doing, working with CloudFlare to identify and mitigate "attacks" as rapidly as possible. But there's no doubt that it cost us a LOT, in terms of developer time, payments to CF, and customer dissatisfaction.

By the way, this was all the more frustrating because we had circumstantial evidence that the attacker was a service contracted by one of our competitors. And if they'd come straight to us to talk about it, we'd have been much happier (and I think they would have been as well) to offer an API through which they could get the catalog data easily and in a way where we don't have to spend all the compute on the value-added stuff we were doing for humans. But of course they'd never come to us, or even admit it if asked, so we were stuck. And while this was going, there was also a case in the courts that was discussed many times here on HN. It was a question about blocking access to public sites, and the consensus here was something like "if you're going to have a site on the web, then it's up to you to ensure that you can support any requests, and if you can't find a way to withstand DoS-level traffic, it's your own fault for having a bad design". So it's interesting today to see that attitudes have changed.

[gwbas1c]

> rather than blocking the bad guys, we'd tag the incoming traffic

> had circumstantial evidence that the attacker was a service contracted by one of our competitors

> we'd have been much happier ... to offer an API through which they could get the catalog data easily

Why not feed them bad data?

[CWuestefeld]

We didn't like the ethics of it, especially since we couldn't guarantee that the bogus data was going only to the attacker (rather than to innocent but not-yet-authenticated "general public").

[IshKebab]

I guess you could have required login to show prices to suspicious requests. Then it shouldn't affect most people and if it accidentally does the worst outcome is they need to log in.

[miga]

Do they change IP numbers so often?

[CWuestefeld]

Oh, lord yes! Frequently they're scraping us from multiple distinct CIDR blocks simultaneously. But we can tell it's the same organization doing it not just because the requests look similar, but it's even possible occasionally to see a request for a search from one CIDR that's followed up immediately by requests for details for the products that had been returned by the search.

While at the same time, because our site is B2B ecommerce, where our typical customer is a decent-sized corporation, it's not uncommon for a single legit user to have consecutive requests originate from different IPs, as their internal proxies use different egress points.

[kristianp]

Stupid question, won't that consume 7000 ports on your own box as well?

[kijin]

Each TCP connection requires a unique combination of (server port, client port). Your server port is fixed: 80 or 443. They need to use a new ephemeral port for each connection.

You will have 7000 sockets (file descriptors), but that's much more manageable than 7000 ports.

[Neywiny]

I think it'll eat 7000 connection objects, maybe threads, but they'll all be on port 80 or 443? So if you can keep the overhead of each connection down, presumably easy because you don't need it to be fast, it'll be fine

[swiftcoder]

7000 sockets, at any rate, but provided you've anticipated the need, this isn't challenging to support (and nginx is very good at handling large numbers of open sockets)

[Orochikaku]

Thinking along the same lines a PoW check like like anubis[1] may work for OP as well.

[1] https://github.com/TecharoHQ/anubis

[hshdhdhehd]

Avoid if you dont have to. It is not really good traffic friendly. Especially if current blocking works.

[CaptainOfCoit]

> Especially if current blocking works.

The submission and the context is when current blocking doesn't work...

[hshdhdhehd]

> Thankfully, CloudFlare is able to handle the traffic with a simple WAF rule and 444 response to reduce the outbound traffic.

That is strictly less resource intensive than serving 200 and some challenge.

[CaptainOfCoit]

Right, but if you re-read the submission, OP already tried that and found the costs to be potentially be too high, and are looking for alternatives...

[winnie_ua]

It was blocking me from accessing GNOME's gitlab instance from my cell phone.

So it mistakedly flagged me as bot. IDK. And it forces legitimate users to wait a while. Not great UX.

[lagosfractal42]

This kind of reasoning assumes the bot continues to be non-stealthy

[lucastech]

Yeah, there are some botnets I've been seeing that are much more stealthy, using 900-3000 IP's with rotating user agents to send enormous amounts of traffic.

I've resorted to blocking entire AS routes to prevent it (fortunately I am mostly hosting US sites with US only residential audiences). I'm not sure who's behind it, but one of the later data centers is oxylabs, so they're probably involved somehow.

https://wxp.io/blog/the-bots-that-keep-on-giving

[swiftcoder]

I mean, forcing them to spend engineering effort the make their bot stealthy (or to be able to maintains 10's of thousands of open ports), is still driving up their costs, so I'd count it as a win. The OP doesn't say why the bot is hitting their endpoints, but I doubt the bot is a profit centre for the operator.

[lagosfractal42]

You risk flagging real users as bots, which drives down your profits and reputation

[swiftcoder]

In this case I don't think they do - unless the legitimate users are also hitting your site at 700 RPS (in which case, the added load from the bot is going to be negligible)

[hansvm]

Once the bot is stealthy (the current sub-thread if I haven't misread) they absolutely do. A couple examples where I've been flagged as a bot for normal traffic:

1. Discord's telemetry was broken on my browser, and on failure they immediately retried. It didn't take many actions queued up on the site before my browser was initiating over 100RPS, on their behalf.

2. Target and eBay still flag my sessions as bot traffic (presumably because they don't recognize the user agent or because I use Linux or something). Target allows browsing their site for a few items before heavily rate-limiting me for a day or so, and eBay just resets my password a day or two after I log in, every single bloody time.

The problem is that from time to time normal users will generate large traffic volumes, and if the bot owner uses many IPs then you're forced to use less reliable signals for that ban hammer (i.e., no single user will be near 700 RPS).

[somat]

xkcd 810 comes to mind. https://xkcd.com/810/

"what if we make the bots go stealthy and indistinguishable from actual human requests?"

"Mission Accomplished"

[HPsquared]

This has pretty much happened now in the internet at large, and it's kinda sad.

[lotsofpulp]

“Constructive” and “Helpful” are unfortunately not out weighed by garbage.

[heavyset_go]

If going stealth means not blatantly DDoS'ing the OP then that's a better outcome than what's currently happening

[SergeAx]

Wouldn't it consume the same number of connections on my server?