Hacker News new | ask | show | jobs
by bigdollopenergy 2340 days ago
Improving performance after the fact is a lot harder than you might think, depending on the nature of your application and it's users behavior, it can be a very steep uphill struggle to realise even small gains. Here's an issue I recently ran into.

When I was tasked to increase performance of a legacy web app, I thought there were so many "quick wins" that I could take a VERY slow application (we're talking 10s long requests for most pages) and get most pages down to less than 3s per request, and at MOST 5s for all of them

One example was permissions being stored in a long "bitmap" string of 1's and 0's that was being parsed on each page load (it was hundreds of "bits" long). The code to handle this was beyond awful complexity wise and added 2s per page load to parse. Some simple refactors and 2s removed from each and every page load. There were a bunch of things like this too, I was on a roll fixing these things.

In my dev environment I set up some performance test harnesses (replaying real world traffic from a representative amount of users) and got benchmarks indicating that i'd increased performance by about 4 times on average. It was still sluggish, but no longer unusable. When I deployed, I found the site was only 1.5 to 2 times as fast. Not quite enough to hit my very modest goal of <5s per request. What the hell happened?

Turns out the site was so slow, but we had so many active users that needed to use it, we were suffering something you'd normally only observe in real life automobile traffic, a phenomenon called "induced demand". When a site is really slow users won't use it unless they absolutely have to. So as you make the site faster they will subconsciously start doing more things things beyond the absolute bare minimum they would normally do. You get stuck fighting some weird kind of "equilibrium" where as the site gets faster you start going below the level of "do what I need and get out" threshold of more and more users who push back by adding more traffic until the site reaches a new equilibrium.

So we made the site 4 times as fast, but the amount of requests more than tripled and negated half our gains. Improving performance on a web app of this nature after the fact is a LOT harder than you might think.