|
|
|
|
|
|
by social_quotient
2842 days ago
|
|
|
Seems like the perfect match for aws lambdas. I’d consider setting the tasks for crawling in to SQS and then trigger the lambdas to go do single crawl functions. For better control over the throttle and concurrency you can leverage dynamodb... I love it for controlling lambdas but not for storage. If you need more power than a lambda then you can do a similar process with EC2. Populate the SQS Trigger the aws lambda to turn on EC2 machine. Consider spot instances to save a ton of money. If you need ideas I’m sure HN readers would be glad to help solution for afar. |
|
|