[arcaster]

Seems like this wouldn't really be useful to scrape js rendered content or any content of "real" value that had any kind of rate limiting or monitoring enabled. Spreading the ip space and making scraping look like genuine user input is a far greater challenge than spinning up a RMQ cluster.

[plantpark]

You are right. But with more codes or tools , it could do this too. It's just a quick demo for distributed crawler. If you moniter traffic of your target website with js rendered content, you will find json file and json api. And what you need next is just the same code in my article.