Hacker News new | ask | show | jobs
by ankimal 4198 days ago
Can you talk about performance? Having written both vanilla non-js scrapers (which get you far enough) and js scrapers for those pesky web pages, I can confirm that our js scrapes took 10-20x the time.

Also, I am assuming dynamically changing IP addresses when you may get blacklisted is built in?

1 comments

The entire anonymous IP infrastructure is indeed built-in. Yes, performance is lower than non-JS-enabled scraping. To counter this we offer a variety of options to turn on/off performance trade-offs. They can be set per job and per page.