[blakesterz]

Are there any defenses I can put in front of my websites that are good for stopping these things? The amount of traffic I see from residential proxies is just killing me. In particular defense against residential proxies.

[Rasbora]

Solutions exist specifically for detecting residential proxy traffic: https://layer3intel.com/tripwire

You can also check if your network is running a residential proxy exit node here: https://layer3intel.com/is-my-network-a-residential-proxy

[jappgar]

The bots used by these proxies are detectable in a few ways. Remember the bot itself doesn't run on the proxy...

There is discernible lag from proxy to c&c node. The individual bots don't have access to a lot of compute, and are sometimes restricted wrt feature set (e.g. proprietary video codecs).

There are a few other techniques. It's a cat and mouse game though. And the bot owners are usually more motivated than you are.

[asdff]

Have a link on your page that would be hidden by users via css. e.g. white text on white background. Have it just abuse compute or do something absolutely stupid for the bots that end up crawling over that link. Hell, just zip bomb them.

[trumpdong]

Make your server so efficient that a few extra requests doesn't bring it down.

Alternatively, if it's the first time the IP is seen and it's a deep linked page with no referer, send a neverending chunked gzip data stream.

[bakugo]

Add a captcha or proof-of-work challenge in front of your website. Those are pretty much your only options.

[asdff]

There's probably ways to go on the offensive too with pages that might be hidden from human users but still crawled by bots.