[mongodbhater]

All the (AWS) technologies used are totally unnecessary. SQS/DynamoDB/Lambda. I can buy a laptop in walmart for $500 and i can do all the scrapping in starbucks wifi.

[rblatz]

Right, it seems like they overbuilt this hacky solution. You are scraping, eventually you just need to subscribe to the data. Why invest that much effort into a temporary solution.

[nunez]

Lambda is needed to get rotating IPs and scale while avoiding browser fingerprinting. SQS takes the results of those scrapes and puts them into a database, DynamoDB. It's a straightforward web scraping pipeline.

[toddh]

Lambda isn't enough. You'll get blocked in a heartbeat. You still need a proxy service.

[atesti]

No, the access google over residental proxy servers provided by packetstream.io

[sdinsn]

Of course its unnecessary. The point is that you can do it in the cloud, instead of on a laptop at starbucks...