| HN Mirror

I'm the other person Ben mentioned :)

The site used RoR for a long time, so I don't want to trash RoR. I still like Rails a lot (and Ruby is my favorite language by far!), but it didn't make sense for where the site was at the time. Also, my background is primarily in ops/sysadmin, and I hadn't use Rails since about version 3.x, and I know a lot had changed since then. I didn't want to spend time re-learning a large framework like Rails for what was a very simple website. So that is a big reason why the idea of using something JS-based that could operate in a serverless container was appealing to me and I lean more on the cloud provider handling resource allocation for spikes...

I had used serverless (Lambda specifically) on a few smaller work projects prior to taking over the downfor site, mostly building simple backend APIs, but nothing on the frontend. I am not a frontend developer at all. As someone that has spent a large portion of their career managing servers, even bare-metal (and being on-call for them), the idea of not worrying about even a misconfigured auto-scaling group was very appealing.

When I took over the site, there were several problems:

1. The site is very bursty, by its nature. We often 4x our traffic or more when a popular site goes down, within a minute or two. The problem the Rails site faced, even with an AWS auto-scaling group, was that it could not respond to those bursts fast enough to bring up more instances before our Passenger slots were saturated and the site would go down. A lot of outages are really quick, so people swarm in, and then they're gone, and our site would be down as a result, which was embarrassing for a site about downtime :)

2. This is related to the bursty nature of our site -- we get a lot of attacks. Because our site makes remote HTTP requests to other sites, people try to use us as a proxy to attack other sites. The Rails version of the site did not cache remote HTTP lookups at all, not even for five seconds, so one request to our site meant one request to a remote site. We were regularly, and understandably, being blocked by a lot of sites which further reduced the efficacy of our website to communicate if a site was actually down or not.

3. The site operated in EC2 instances, so the cost was very high for what it was doing. It varied between $600-$800 because of the attacks and how many times the auto-scaling group would bring up more instances. As most everyone knows, too, the bandwidth costs at AWS are high and we would regularly face multi-Gbps attacks. State was stored in a Postgres database, which was also expensive, but not really needed given the simplicity of the site.

4. Page load speed. Google regularly complained about site speed because Passenger requests were being saturated intermittently, and there were too many dynamic requests being made on every page load, and the cache rate was awful.

5. One of the glaring problems with the previous site was that the remote HTTP request happened inside of the frontend Rails application in the foreground, so every page load, even by bots, triggered that request. The request now happens client-side, which greatly reduces the amount of remote HTTP requests we perform (and subsequently have to wait on finishing). This was a big reason why Passenger requests would be saturated frequently, because our site would be waiting for a remote website to be down in the foreground, on websites that are, as you an can imagine, likely going to be down or hanging for many seconds. I think this is why you see a lot of our competitor websites regularly go down during large outages -- their foreground processes are saturated waiting for down websites to timeout.

The previous developer made most of the Rails version of the site, but they did not have much of an operations background. It ran in Elastic Beanstalk, too, and I didn't like that dependency if I ever wanted to move it somewhere else. I maintained that version of the site at first and tried to keep it online and stable, until I had enough (especially being paged in the middle of the night) and decided to rewrite it from scratch and dramatically simplify it. I did not want to ever be woken up in the middle of the night for a side project like this, and I also hate seeing a site I run have downtime or perform poorly :) I am normally against complete software rewrites, having experienced that pain before in the past, but I didn't see much to salvage and the site was very simple, so the rewrite did not take very long.

The first serverless iteration (still NuxtJS) was entirely on AWS, which was definitely an improvement over what we had in page load speed, cache rate, management time, and cost. But we still faced regular distributed multi-Gbps attacks that AWS was obviously happy to charge us for. I tried to manage some of these attacks with the AWS WAF service, but it felt very much like a beta service, barely ready for production usage. It did help prevent a lot of nasty stuff getting to our backend, which was more expensive than the WAF, but it was not pleasant to use and not flexible.

I had considered setting up a pool of Nginx or HAproxy machines myself, possibly with mod_security, to help mitigate some of the smaller frontend attacks and prevent our backend from being saturated, but I decided to move the site to Cloudflare instead because the cost was dramatically better. That also fixed the problem of stopping both high-bandwidth attacks and layer-7 attacks that would saturate our backends. Not having to pay per-request and the high bandwidth cost in AWS saved us a lot on infra cost instantly; it also gave us more flexibility to perform JS/captcha challenges to obvious attackers and prevent them from getting to our backend. We still get attacks today, but they are much easier to manage and modify the rules as needed, and I can do that from my phone easily instead of having to modify a config file. Cloudflare also gives us access to a great CDN, without having to pay AWS Cloudfront pricing.

A lot of the problems with the Rails site could likely have been fixed by moving the frontend to Cloudflare and working on the cache rate and attack problem to better protect the backend, but I knew I was going to be the only person working on the site, so I wanted a stack that I was comfortable with and could build upon. I still believe Rails was overkill for a site of this simplicity. I also hadn't worked with JS much at that point, as I'm not a frontend developer, so it was a relevant learning experience and it also was a nice transition to using JS more for serverless applications. I had also considered keeping all of the backend operations to a Rails app, but move the entire frontend to a static site.

This is not to say serverless is without problems, but I do find that people denigrate serverless a lot on this site, or dismiss it as some fad. I'll definitely be writing more about that on our blog when we launch that, but I always like to use the appropriate tool for the job, and I still think serverless was a good move for us based on how our site works and me wanting a "hands off" approach to operating it. And I think the results speak for themselves in increased performance, much lower cost, almost no active management time, and we regularly hit 100% uptime.

If you have any other questions, let me know! My email is also in my profile if you want to contact me directly.