[bdcravens]

Selenium is great, but seems to be easier to detect and block.

[dwd]

Selenium lets you add random delays between your actions which could help avoid triggering a firewall to block you.

Good practice anyway so you don't overload the site and find your logs empty or full of gaps.

[bdcravens]

Good approach, but advanced Selenium detection goes beyond heuristics. Selenium injects JavaScript into the page to function, and the presence of this is how Selenium is detected.

[dwd]

Interesting, I've worked on both sides of scraping and protecting content but hadn't really considered checking for JavaScript frameworks as a trigger. I'm assuming this is something you could configure in a F5 that also injects its own JavaScript?

Randomising field names, seeding hidden bogus data and messing with element order was more what I would look at once a persistent scraper was using enough IPs to get around rate limits.

[bdcravens]

I believe you could recompile Selenium with different names to get around it, but since Puppeteer uses CDP baked into the browser, no injection is necessary, bypassing alot of this.