[sreque]

Assuming the author's tests are single-thread, I'm pretty sure 1024 MB doesn't give you a full CPU core on Lambda. I could be wrong though; I haven't payed attention to Lambda in a long time. Last I remember it was 1.5 GB that gave you a full core. This alone makes the comparison between a mid-range server and Lambda unfair, not to mention the differences between language runtimes.

That said, if you are using Lambda and expecting to not pay extra you have somehow been mislead. Lambda is definitely more expensive per cycle than managing your own instances, and I doubt that will change any time soon.

[hobls]

Anyone using Lambda should _absolutely_ do load testing with different memory configurations. You will get different results, and should analyze what is best for your application.

When calculating the overall cost of managing your own instances you should also include time spent by your engineering team. There are particular tipping points in terms of overall requests per second at which point you'd save money by moving from Lambda to something like Fargate, and then even farther above that, you're better off using EC2. And then even above that, you should be running your own instances in a colo space. (And then at some point you should probably be building your own datacenters, and then at some point you should start colonizing the moon, and then... you get the idea.)

[chx]

> And then even above that, you should be running your own instances in a colo space

Why are people jumping from EC2 to colo and skipping dedicated servers? Mystery of my life. We were running the 75th largest site in the US some years ago (as measured by Quantcast), ran the numbers and colo was ridiculously expensive and way more troublesome.

[mmt]

> colo was ridiculously expensive

That's slightly surprising to me, unless you were limiting yourself to a particular geographic market for colo that was unusually supply-constrained at the time. (The SFBA during certain years comes to mind).

Sometimes rented servers, for a specific use case (or if the provider is overstocked on a particular model) are a great deal, but I've never seen that at the high-performance end of the spectrum, if they even offer such a model in the first place.

For the average and middle-performance cases, though, for truly comparable servers and connectivity (internal and external, which can be tough to find in the first place), I found rented servers to be moderately more expensive than colo plus buying hardware amortized over 3 years [1].

> and way more troublesome

This one, is the mystery of my life. You mention "magic smoke" downthread, but I've only experienced that once in my entire career and that was with proprietary hardware 2 decades ago.

Conversely, my experience with rented servers is that when there is a hardware problem, other than obvious failure of a replaceable part, "troublesome" doesn't begin to describe it.

[1] yes, including all the costs like installation/rack/stack, network ports, spares, etc. They're not de minimis, but it's maybe a few extra percentage points on the overall cost.

[hobls]

Oh, I was defining "EC2" as everything from small shared virtual boxes to EC2 bare metal. I suppose on-site dedicated servers is definitely a point I left out of my hypothetical list though! I used to work at an unnamed company that had a server rack plugged in between the kitchen and the ping pong table. Though I never witnessed such a thing, I'd be shocked if there was never an outage due to a particularly enthusiastic game.

[chx]

Not on site, no, renting them from some big provider. The advantages are huge: if the magic smoke gets released getting a replacement on line is Someone Else's (TM) problem. If you find you need to grow, you release the one you rented into the pool and the provider will be able to rent it to the next schmuck. EC2 bare metal is very expensive. In general, I am very cloud skeptic. I found the bare metal being a much better price/value for almost all websites. People are writing these complex apps that scale across database clusters and I am like... YAGNI. One database server (or a two for a hot spare) and maybe separate few web frontends is almost always enough.

[hobls]

I’m sure that makes plenty of sense for many applications! I’m very fond of some of the benefits of cloud infrastructure, but I’m sure you’ve heard the pitch.

[philipodonnell]

Are there any tools that allow you to load test Lambda services with different memory configurations?

[hobls]

Not to my knowledge. I'd just do something simple by changing your CloudFormation (or Terraform or whatever) templates, deploying the new config, and then running another load test. Could also just spin multiple different versions of the Lambda up, but at some point you need to start getting creative to actually generate enough traffic and running multiple load tests at the same time becomes a pain.

When it comes to load testing tools I like Vegeta[1], personally. (Though I've also used some much more complicated proprietary tools when testing at great scale.)

1: https://github.com/tsenart/vegeta

[reilly3000]

In Serverless Framework you can specify memory per function so it stands to reason making multiple copies at different memory levels would be pretty easy to test, even concurrently.

[hobls]

The real fun starts when you need to start spinning up a distributed load testing fleet to actually generate enough load. :P

[zlynx]

Has anyone done research on cooling datacenters on the Moon?

[NathanKP]

It's not quite that straightforward. Lambda is more expensive per cycle if you are capable of keeping an instance fully utilized and getting every cycle out of that machine. After all if your Node.js or Go code takes say 20ms per request and can handle many concurrent requests you can squeeze a LOT of requests per second from a single instance.

But many workloads don't have that high of a request volume and can't actually make full use of an instance. If you have a small API or service that gets one or two requests every few seconds then paying for a 100ms chunk of Lambda execution time every couple seconds is going to be much cheaper than reserving an entire instance and then not being able to get good utilization out of it.

The tipping point is whether or not you have enough workload volume to keep an instance busy at all times. So for example password hashing in the article above. Because password hashing is deliberately CPU intensive it is very easy to keep an EC2 instance busy with even a low request volume. For a good hashing algorithm with lots of rounds its not uncommon to only get 10 authentications per second per core, because the algorithm is deliberately designed to be CPU heavy. So if you process more than 10 auths/sec then its probably cheaper to put the workload in a container that runs on an instance because you can keep that instance busy.

But if the same service is only handling one or two password hashes every minute, then you can save money by only paying for 100ms increments when an auth request arrives, and stop paying when there is nothing to do.

[sfeng]

How does that compare to the cost of cloudflare workers?

[NathanKP]

No idea. Given that a secure password hash will probably take about 100ms-200ms of execution time to calculate that would fall under Cloudflares "custom pricing" tier that you need to call them and negotiate.

The baseline Cloudflare worker tiers are limited to less than 5ms, less than 10ms, and less than 50ms, which isn't going to be enough time to calculate a 12 round bcrypt for example.

[zackbloom]

I have a Worker running bcrypt here: https://cloudflareworkers.com/#4addaef33b10b6a58954ffbb310e7...

Based on this code: https://gist.github.com/zackbloom/c0064838cbf85e7b81df9d4690...

That means it would cost you $0.50 / million requests. AWS Lambda would be $1.84 / million, $3.50 / million for API Gateway, $0.40 / million for AWS Route 53, and various other charges.

[NathanKP]

Technically CloudFlare workers is better compared with Lambda @ Edge no? I don't think you'd be using API Gateway, instead CloudFront right?

[zackbloom]

Originally we were thinking about it in those terms (as an alternative to Lambda@Edge), but based on our recent results I am happy to have people compare it to Lambda as well. Lambda with API Gateway is at least eight times more expensive than Workers, and only runs your code in one location instead of 151. Unless you're using Lambda to do something ultra-Amazon-specific (like process S3 changes), I don't see why it would be the better choice.