Hacker News new | ask | show | jobs
by ammario 18 days ago
There are PoW approaches that even the playing field between data centers and desktops. RandomX is my favorite.
2 comments

For what it's worth I'm working on hashx support. It's just going to take a bit to ship while I do browser testing with broken browser configs.
Interesting. How do they tell the difference between legitimate and forged ip owner records?
It's not about traffic identification at all, but rather a hashing algorithm that is deliberately resistant to parallelization and GPU/ASIC acceleration, which shrinks the gap in solving speed between the fastest systems (i.e. datacenter-class compute resources) and typical systems (e.g. the CPU in your smartphone or laptop).
Uh, is it resistant to parallelization across multiple sites? Because that's the situation for the scrapers. They're not trying to solve a single PoW challenge across many cores.
Typically, the machine doing the content processing, including solving PoW, is the centralized "control" node described in the article, not the machines who's IP addresses are being used. In typical residential proxy networks, the residential proxies are exposed to the customer (the person paying for and using the proxies) as just SOCKS5 addresses, and no computational power from those compromised devices is made available for the scraper besides that used to power the SOCKS5 server itself, the customer is just paying for the transport and address (and indeed, is often billed on either a per-GB or per-IP basis).

In effect, if the customer (the entity paying for and using the proxies) wants to solve PoW challenges through those connections, it is indeed the customer who must pay that compute cost, not the compromised devices.

Note that this is the case for a majority of, but not all, residential proxy networks, which often are built through quasi-voluntary distribution channels, including SDKs included in otherwise legitimate mobile applications distributed through Apple's App Store and Google Play.

These distribution channels tend to be categorically unavailable (or at least unreliable) for true RAT-style malware that enables remote operators to dynamically assign arbitrary computational workloads to client devices.

This isn't to say that true botnets built with actual malware delivered through either software exploits, phishing attacks, or watering hole attacks don't also perform as residential proxy networks, but such categories are a relatively small subset of all residential proxy networks, and there are much higher ROI malicious activities to be performed on these devices rather than serving as relatively mundane traffic networks for scraping.

That's a completely different question, your claim was about parallelism.
Ah, I see what you're getting at. Yes. You can think of any given computer aa having a fixed amount of compute budget for these types of acceleration-resistant hashing algorithms. Let's say the scraper can perform 10,000 hashing operations per second total on their machine, and needs an average of 1,000 hashing operations to solve the PoW. It's a minor detail, but note that these PoW challenges non-deterministically vary in the number of hashing operations needed to produce a valid hash, not dissimilar to bitcoin mining, where a hash with a certain number of 0s prefixed is sought, and the scraper essentially has to brute force through all possible inputs until an input that produces a valid hash is found.

In a well-designed PoW systems, there is a per-site prefix or suffix that is required to be prepended or appended to these random inputs, and it may change not only between websites, but even between PoW sessions on the same website, and should not be predictable - only being disclosed to the client at the time the PoW challenge is issued. In such a case, the scraper cannot simply precompute a bunch of valid hashes that work across multiple sites, nor a bunch of valid hashes that will always be good for even one site, the scraper operator will need to compute these hashes (with a limited budget to do so) upon initiating each PoW session.