|
|
|
|
|
|
by djm_
3508 days ago
|
|
|
Agreed for some things but a massive limitation of Lambda with its timeout of 300 seconds is that it cannot be used for long running tasks. If this feature lands it sounds like it will give the ability to run any container arbitrarily and pay by the second - this would be huge for things like web scraping and other tasks that don't occur all the time yet still come with all the pain of server maintenance and uptime fees. Imagine the scenario where you have a web scraper that runs once a week for 3 hours. Ideally you only want the machine to be on for those 3 hours, and to only pay for those hours of usage; but you also don't want the hassle of writing all the scripts that go with creating and deleting a cloud machine, mostly because those would also need to be hosted somewhere. For that kind of use case there is little out there as far as I'm aware. |
|
|
Look for ways to divide-and-conquer your lambdas into smaller parts. If you need to run some logic for every record in some table, give N records to a single lambda (where N is some small number which doesn't make the lambda take anywhere close to 300s).
You can orchestrate this workflow with several different AWS tools which exist all over the spectrum of cost and ease-of-use.
Easiest is definitely just having the master lambda directly invoke the other lambdas with InvocationType:Event.
SNS is another easy option. Lambda(master)->SNS(per N records)->Lambda(splinter). Downside is that you'll probably completely blast out your global AWS concurrent function execution limit pretty quickly because you have no control over how quickly SNS will trigger your functions.
Kinesis is a more powerful option. SQS also has potential, but you can't directly trigger a Lambda from SQS. One pattern I've seen used is to have a CWEvents cron trigger a lambda every M seconds to read N records from SQS. Depending on how consistent your workload is, this might make sense because it gives you really fine-grained control over that ratio between "how quickly will my jobs be processed" and "am I approaching my AWS global account limits". But if your jobs are really disparate you'd be invoking lambdas all day to do a whole bunch of nothing 90% of the time.