Hacker News new | ask | show | jobs
Cloudflare's new Rate-limiting. Beware
19 points by gopi_ar 3190 days ago
Early this month, our devops engineer went on leave. On the same day, usage spiked on our opensource API. Disabled tokens were making 1000s of rps causing heavy load. Not wanting to disturb our devops engineer, we tried IP blocking in CF, which didn't help because requests were coming in from 100s of IPs (probably app users).

We then saw - on the same page - CF's new rate-limiter. It seemed great, and I think it said '1 free rule'. We quickly set it up to rate limit to a few rps. This also did not work, because there were multiple tokens making requests from 100s of IPs. We finally ended up filtering those tokens out on NGINX.

Fast forward to yesterday when we got the bill. I don't usually open them because it's always $40/month. This time we added their LBs (they don't support session stickiness, so again, beware) so I was curious about the charge.

$876.

$90 for the LB, fine. But the clincher? $721 for the 'rate-limiter'.

Here's why we have an issue with this:

1. Rate-limiting did not work for us. 144 million requests passed through. 28 requests were rate-limited. 28. (facepalm)

2. The pricing is misleading; In the heat of things, I only remember it saying '1 free rate-limiting rule' and missed their note on usage pricing. Yes, silly of me to assume that CF would continue their claim-to-fame as the single unmetered vendor. You have to click the 'usage' link nearby and read the blog post to understand pricing.

3. No billing alerts whatsoever. When usage is over 20x of a user's monthly charge, you'd expect some form of an alert. To put this in a USD -INR context, that's ~2 month's salary for our devops guy.

We absolutely love CF and have been evangelizing them since we started using them 2 years ago. I've reached out to support and their first response was to say there'd be no refund. Let's see how this plays out. :-)

In the meantime, if you're using CF please check your usage to make sure you're not running up 20x your monthly costs.

3 comments

Interestingly, it's their 7th birthday this week and they announced 'unmetered mitigation'.

https://blog.cloudflare.com/unmetered-mitigation/

"So today, on the first day of our Birthday Week celebration, we make it official for all our customers: Cloudflare will no longer terminate customers, regardless of the size of the DDoS attacks they receive, regardless of the plan level they use. And, unlike the prevailing practice in the industry, we will never jack up your bill after the attack.

Doing so, frankly, is perverse.

We call this Unmetered Mitigation. It stems from a basic idea: you shouldn't have to pay more to be protected from bullies who try and silence you online. Regardless of what Cloudflare plan you use — Free, Pro, Business, or Enterprise — we will never tell you to go away or that you need to pay us more because of the size of an attack. Cloudflare's higher tier plans will continue to offer more sophisticated reporting, tools, and customer support to better tune our protections against whatever threats you face online. But volumetric DDoS mitigation is now officially unlimited and unmetered."

:-|

Maybe the OP can talk to them with this article in hand and see if they can get some money back?

It never hurts to ask.

I don't want to kick a guy when he's down but I think the key takeaway here is "our devops engineer", singular.

Maybe s/he is just a developer playing at ops anyway and it wouldn't have helped but if there is literally one person even slightly familiar with your infra and ops, you have a problem unless that person is Mr Data from star trek and never goes on holiday, gets sick, has a night out, or heck has a weekend.

Depends on the size of the company and how complex is their infra. May be the founder can rollup his sleeves and get things done when needed. Nothing wrong if that's what it takes for the business to survive.
DevOps is about shared responsibility between dev and ops teams. One guy with the title "DevOps Engineer" responsible for everything is doing it very wrong.
Is that true for small companies, too? Does the financial department have to be more than one person, too? What about sales? Logistics?
Those departments aren't responsible for maintaining the infrastructure your company relies on and customers expect to access 24x7.

Having just one person who can manage your ops is like having a single cop and wondering why crime goes up when he's off duty.

Some of those can be, though. Logistics definitely is...
It's just a matter of scale; DevOps is a luxury at our scale. That said, we're getting other devs trained in basic ops to prevent such things..
They refunded the 'rate-limit' charge. Phew! Thanks to @jgrahamc, their CTO, for weighing in.