[willsmith72]

I always love a story about a successful strangler pattern migration.

I do wonder how they came to Lambda though. I love it for small workloads and highly variable demand services, but something like Treezor you'd think has relatively flat and high demand. The cloud cost for Lambda would be much higher than running the equivalent compute, even with something also highly scalable like ECS.

[ceejayoz]

We use Lambda (via Laravel Vapor; https://vapor.laravel.com/) for a big analytics batch processing job every night; it spawns hundreds of thousands of individual jobs that make various API calls, but only for a few minutes. For the rest of the day it's doing very little.

I'd imagine a bank has quite a few of these sorts of bursty bash processing needs; ACHes, end-of-day reconciliation, etc.

[ndriscoll]

At my last job, we had an architecture that ran hundreds of thousands of individual jobs with internal API calls for nightly account updates and reporting work, and it was a major battle to finally get it replaced with some simple SQL scripts which ran orders of magnitude faster while being a lot more reliable.

As I understand it, banking commonly uses JVM where you can easily do async or multithreaded processing and task scheduling, so lambda doesn't offer much over ECS, and the timeout could be a real killer.

[jackconsidine]

What's your experience with Vapor? We played around with it and thought about switching from Forge, but it was too complicated for our codebase. I love the idea though.

How expensive our your cloud costs? CI/CD pretty good?

[ceejayoz]

Overall quite good. We use their Docker container support (to get ffmpeg, mostly) and it's been pretty smooth running throughout. I'd previously used a lot of Heroku so the habits developed there paid off in our codebase for Vapor; only a few tweaks were really needed.

Cost-wise we're pretty bursty, so it's saved us quite a bit. Lambda runtime is money, so a little bit of performance optimization can go a long way. CI we run through Github Actions; an environment secret plus `vapor deploy` and it's on its way up after a successful run.

Wasn't a fan of Forge; for that style of things I'd use Ploi nowadays.

[theshrike79]

Lambdas are amazing at clearing queues.

You can have a single queue gathering workloads and then have Lambda functions grab a bunch of tasks, handle them, grab more etc. until the max runtime is met and they're automatically rebooted.

It's also pretty trivial to add more processing capacity when the queue goes over a specified limit just by allowing for more concurrent Lambdas.

And when there's nothing to process, you've got one idling Lambda doing nothing and costing nothing.

[me_graf]

I did some fintech work as a junior / kinda non junior dev. This is correct. I think 90% of the work was taking pictures of checks, and setting up batch jobs. all batch: - money movement (between accounts) - transfers - wires - etc.

even the 'non' batch things were done in batch (even if it's do now, and only 1).

[iLoveOncall]

> I do wonder how they came to Lambda though

Probably misplaced hype about being fully serverless.

As you said, ECS would be much more suitable while also addressing the issue that they had (spikes in traffic).

Lambdas behind API Gateway sounds like a good idea and even looks like one if you don't have experience with ECS and don't do a proper cost comparison analysis.

My team uses ECS' autoscaling to scale up to tens (or even hundreds) of thousands of tasks in minutes without issues.

[Supermancho]

> I do wonder how they came to Lambda though.

Probably some security certification that AWS Lambda comes with. I worked at Cambia Health that used Lambda (Javascript) for dealing with customer mobile data. It was a nightmare solution (cost and complexity) for what should have been a simple fan out queue and process system. They hired me to help fix it. Every other solution was a flat no. AWS Lambda was was HIPPA compliant (for whatever that's worth) and some other security assurances that I don't remember, so I was basically just another developer (with a moron for a manager).

[mnapoli]

Their clients have very variable traffic patterns because they are in very different industries. They have traffic spikes at lunch, or at the end of every month for example:

> infrastructure must be able to scale and be resilient to accommodate various usage patterns. Whether it's a luncheon voucher transaction spike at lunchtime or a monthly batch of transactions by corporate clients

[cogman10]

This was my thought. Lambda works well for infrequent tasks. But hosting an API? That seems like it could get expensive pretty fast.

[deleugpn]

It's cheaper than most options where you need to pay a human to manage something with the same level of scalability and security that AWS provides.

[ndriscoll]

Lambda scalability for an API is awful; it can't run concurrent requests (of course PHP largely can't do this either). An ECS service with pretty much any other language (for async/multithreaded runtimes) will scale far better.

[cebert]

A single lambda can only handle one request at a time, but you can scale up 1,000s of instances if needed.

[ndriscoll]

A single minimal ECS instance can handle thousands of concurrent requests with an async runtime without running into so many cold starts that your p25 time is in the hundreds of ms.

1000 lambdas with 1000 database connections is also going to be a horrible way to do things on the backend. Add in the cost of RDS proxy for lambda and the ECS solution becomes even better.

[WatchDog]

ECS fargate is pretty easy to scale, and much more cost effective.

I personally don’t think running on EC2 is particularly difficult to manage, and it’s even cheaper again.

[Sohcahtoa82]

Lambda is $0.0000166667 for every GB-second, which works out to ~$43.20 for 30 days. For a little more ($49/month), you could get a c6g.large and get two dedicated CPU cores (ie, not the shared burstable t2/t3) and 4 GB of RAM.

So yes, very expensive fast if you get much traffic at all. That c6g.large should easily be able to handle dozens (if not hundreds) of requests in parallel, especially if many of those requests are going to have to wait for results from a database.

If you have more than ~800 hours of Lambda execution time per month (30 days being 720 hours), then you're better off with the c6g.large. The only reason to choose the Lambda is if your traffic is extremely bursty and the c6g.large can't handle the load.

Of course, this is all assuming you're using a 1 GB Lambda. Likely, your Lambda needs less, in which case the amount of traffic needed to make the EC2 more worthwhile higher.

[deleugpn]

Then your EC2 instance suddenly stops working and your downtime cost you a fat contract that makes the cost of serverless a pocket change.

If you want to compare AWS Lambda with EC2, you need to bring in AWS Load Balancer, Auto Scaling, Security Upgrades for the OS, Healthcheck and many more.

Yes, Lambda is not needed in a lot of context, but if you have the expertise to use it and you want to provide enterprise-grade compliance, it's a walk in the park.

[willsmith72]

Everything you listed is why I mentioned ECS though. To me it's a perfect in-between.

Bringing the long-lived server back also makes things like long-lived connections, memory caching, cold starts and database connections a non-issue again.

[cebert]

With Lambda you’re in three different Availability Zones by default, so you also need to factor running your proposed EC2 solution in 3 AZs too.

[Sohcahtoa82]

Good points all around.

The costs certainly are not as cut-and-dry as I originally expressed it.

[ndriscoll]

2 cores and 4GB RAM should have no problem handling thousands or tens of thousands of concurrent requests on the JVM. Lambda's documented concurrency maximum is "tens of thousands"[0], which means an 8GB m6g.large or ECS instance can probably handle larger bursts than lambda can. If you had extreme bursts, you'd also be having almost every request hit a cold start.

[0] https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-...

[pfych]

I've been working with Lambda across a few large projects for their API's, with each endpoint being its own lambda.

This has been scaling really well for us and affordability wise it makes the development and testing phase basically free, with the final deployed costs almost never exceeding double digits monthly. This is for multi-az + multi-region deployments too.

[tomrod]

I'd be more worried about decimal data types being interpreted as floats. That could get gnarly fast on the wrong language runtime, and is partially why Cobol programmers are still in demand in the financial sector.

[chx]

the beauty of bref is you work with the same php engine as before. I am a (very) long time PHP developer, my primary work is still Drupal so the ability to make a small serverless app in PHP was very handy for me. My app implements a webhook and I feel Lambda is made for this: the script is small and quite fast, it checks the payload and puts an item into SQS as appropriate. Then another infra can slowly empty the SQS queue. The problem here is the load is insanely spikey -- sometimes tens of thousands of requests in a very short time frame, sometimes nothing. Directly writing this into a transactional database would require very costly infra. The Lambda-SQS model smooths it out. It's fine if some changes take minutes or even hours to be recorded.