[doc_holliday]

Lambda has a long way to go, it's a promising product but if you want to develop anything for it, it's a bit painful and archaic.

I've had issues with the temp memory. It is completely unreliable, you have to store files when processing in /tmp/ and for whatever reason, reading from this results is patchy. I had to resort to storing every file I process in the RAM.

The tooling is only just taking off now, 6 months ago simulation was painful, at least there is AWS SAM now and docker support.

However despite the pain, it's giving me a solution for scalable video encoding that is very cost competitive and I don't have to worry about submitting jobs and monitoring resources.

[davidgf]

AWS SAM is ok, but I encourage you to take a look at Serverless framework https://github.com/serverless/serverless, which eases a lot managing the whole infrastructure. There are already some applications running completely on Lambda + Serverless, such as this one https://github.com/microapps/MoonMail

[doc_holliday]

Thanks, serverless looks interesting.

[jackpeterfletch]

Its not really for stateful services though right?

[doc_holliday]

Video encoding would be stateless. There is no reliance on previous runs / state.

Lambda is designed to be used for tasks like that, such as resizing images, parsing file etc.

[jackpeterfletch]

In a semantic sense it is stateless.

But what I mean is that its clearly not designed for you to be dropping files onto the local disk.

Video encoding is kind of a corner case in that the size of the data really stops you from doing the whole thing in memory, though the persistance isn't needed in between runs.

Probably what you want to look into is attaching an EBS volume to the job, and doing the work from there.