[sam0x17]

Would it be practical to use puppeteer / headless chrome from within a Google Cloud Function to do this? Then you could do literally millions of profiles per hour. The only requirement is that it never takes more than 540 seconds for any one profile, and that you could work around if you write in the ability to export the execution state and send it off to a new function invocation if you are close to the deadline. This seems like a much easier problem if you can get out of VPS land and into lambda/cloud functions since its something like $0.00001 per invocation.

https://cloud.google.com/blog/products/gcp/introducing-headl...

[gitgud]

That's amazingly cheap, but would each puppeteer instance have its own IP address? They're being rate limited on each IP.

[sam0x17]

I would think you get a random IP as they have millions of worker nodes, but I don't know.