Hacker News new | ask | show | jobs
by b0a04gl 354 days ago
why is bot detection even happening at render time instead of request time. why can't tell you’re a bot from your headers, UA, IP, TLS fingerprint. imo making it a surveillance. 'you're a bot, ok not just go away, let’s fingerprint your GPU and assign you a behavioral risk score anyway'
1 comments

It's really hard to detect it at request time. It's practically trivial for an attacker to fake headers to resemble a real browser.
You absolutely have options at request time. Arguably, some of the things you can only do at request time are part of a full and complete mitigation strategy.

You can fingerprint the originating TCP stack with some degree of confidence. If the request looks like it came from a Linux server but the user agent says Windows, that's a signal.

Likewise, the IP address making the request has geographic information associated with it. If my IP address says I'm in Romania but my browser is asking for the English language version of the page... That's a signal.

Similar to basic IP/Geo, you can do DNS and STUN based profiling, too. This helps you catch people that are behind proxies or VPNs.

To blur the line, you can use JavaScript to measure request timing. Proxies that are going to tamper with the request to hide its origins or change its fingerprint will add a measurable latency.

None of these are conclusive by any means. The IP address check you mentioned would mark anyone using a VPN, or English speakers living abroad. Modern bot detection combines lots of heuristics like these together, and being able to run JavaScript in the browser (at render-time) adds a lot more data that can be used to make a better prediction.
> If my IP address says I'm in Romania but my browser is asking for the English language version of the page... That's a signal.

jesus christ don't give them ideas. it's annoying enough to have my country's language forced on me (i prefer english) when there's a perfectly good http header for that. now blocking me based on this?!

Anubis does it pretty decently.
Anubis is not meant to fully stop bots, only slow them down so they don't take down your service. This kind of bot detection is meant to prevent automation.