Hacker News new | ask | show | jobs
by exikyut 1457 days ago
The main issue with all apps like this is not the reverse-engineering, which is relatively straightforward, but the fact that HN sits behind Cloudflare, with the WAF and ratelimiting enabled.

It's not possible to achieve capability-parity just *reading* HN as the API does not return comment karma, which is used to sort the view. (I think this is the subtler half of why HN reader sites never catch on - the interesting comments are never at the top, so the presented content is very noticeably that much more boring.)

So a while back I tried building a simple scraper component (an internal API, I guess) that would use Firebase for realtime updates, and strategically poll the site (say one request every 3-5 seconds) to get the current sort order. I never got to the Firebase part because I crashed into Cloudflare while fiddling around tuning the ratelimiting.

So yeah. You need full Chrome (to satisfy whatever Cloudflare's WAF throws at you) and a captcha-solving subscription (because the WAF will get the flyswat out when it sees your datacenter IP) to be able to do this properly - especially if you want to APIify this sort of thing.

2 comments

I recently wrote a HN reader and decided to forgo handling comments for reasons you mention, including the obscurity of comment karma scores. I focused on the story browsing side of things, and the links to comment sections take you straight to HN’s comment page.

Link to my app if anyone’s curious. Red question mark in upper left corner of page goes to its “about” page:

https://www.thnr.net

Actually if I recall correctly, even though official api doesn’t show Karma value of comment, the children comment IDs (kids is the field name) of each comment are actually sorted by karma (or whatever method HN uses to sort on server side)