Hacker News new | ask | show | jobs
by tonymet 1409 days ago
Obviously scraping logic using puppeteer, but there are many other tooling aspects that are critical to bypass bot prevention.

one is signature / fingerprinting emulation. It helps to run the bot in a real browser and export the fingerprint (e.g. UA, canvass, geoloc etc) into JS object . Add noise to the data too.

Simulate residential IPs by routing through a residential proxy. If you run bots from cloud you will get blocked.