Hacker News new | ask | show | jobs
by eloff 2130 days ago
There's always a comment here on HN wondering why company X has so many engineers for such a seemingly simple product.

The answer is, unsurprisingly, not that they're terribly inefficient. Rather it's that running a popular service at scale is hard, the long tail of features is really a long tail, and a lot of effort goes into even simple things because small differences really add up when multiplied by large N.

5 comments

This is a big part of it, but I wouldn’t downplay the inefficiency too much. A pattern I’ve personally seen:

* Small team of 5 devs and maybe 1 manager working on some large area of the product is very efficient

* Startup is growing users/revenue very fast, attracts big VC investment

* Major hiring, a year or two later that same area of the product has 30 devs, 5 product managers, 5 dev managers, 3 designers, a manager of the dev managers, a manager of the product managers/designers, and a manager of the dev and product managers

* There’s too many cooks in the kitchen, nobody feels empowered to make decisions, meetings multiply like crazy, every decision takes exponentially longer to make than before

* Instead of planning 1 sprint ahead, you start doing quarterly plans, 3 year visions, customer facing roadmaps, etc. “Predictability” becomes more important than productivity

* Teams know that only predictability matters, so they start making really conservative estimates, just to be safe

* If you have 2 months to do 1 month of work ... it’ll take 2 months. And this starts becoming the new norm

* Plus, you start spending more and more time planning, “scoping out epics”, etc., to get “better” estimates

You get the idea - pretty soon that 45 person team ships not much more than the 6 person team used to. Nobody is purposefully dogging it, or trying to go slow, but it happens when companies grow unless you’re really, REALLY good at fighting it. And few companies are. Everyone is BUSY, but they’re busy doing quarterly plans, “full potential analyses”, change management, meeting with stakeholders, working on all sorts of checkboxes to get all sorts of security certifications to sell to one or two big government buyers, working on “business development” projects that users don’t care about, etc.

It's also about the kind of features you implement. At the start you implement all the easy features, that's no problem. Once you mature and want to get B2B sales, you need to start supporting all kinds of connectors to legacy Enterprise software, networking stacks and environments. That's much harder and much more frustrating than building a simple ticketing system.

As an example, supporting OAuth is easy. But making sure your software works with all kinds of on prem LDAP for authentication is much harder.

Yeah totally. I think most feature work at a B2B SaaS company can be classified as either trying to drive retention, or trying to drive new sales. The “new sales” features are often focused on one or a handful of potential large customers with very specific needs, who won’t buy without feature X. This kind of work is basically invisible to almost all of your customers, but is necessary to close the largest deals.
I would argue that some of that baggage doesn't make you less efficient. I mean it makes you less efficient at the features per hour metric, but what about the business value metric? If you have meetings to really think things through and plan, maybe whole areas of sterile development work are avoided and that is valuable.

Remember there are businesses that do well writing no code at all! Code or features isn't what it is al about, but in a tech company it is somewhere between nothing and everything.

Yeah, fair point. This is more an answer to “how can you have such a large product/development team, and ship so few new features”, as I believe that’s a natural consequence of having a larger business, with far more communication overhead, far more focus on planning and predictability, etc. That’s not necessarily bad for the business, but it does mean you have a lot of people shipping a relatively small number of features, especially compared with your velocity in your startup days.
Yes companies definitely get more inefficient as they grow, for all the reasons you cite and more. I've experienced it first-hand.

Adding an engineer to a team slows down the whole team a little. There comes a point where adding head count actually results in less productivity overall. This is typically countered by splitting up work into various teams with minimal interaction between them. Like Amazon's famous two-pizza team strategy. This is the main attraction of microservices, each team can own their own piece of the code and change it and deploy it without (ideally) having to consult with anyone else.

There's always a mix between "now we're at scale and need to solve this really hard data sync problem" and "let's build an internal slack clone because we can".
Ouch, data sync his close to home for me. That was my first job. Not an easy problem at all.

Tip for those of you with the same problem, if you can, try to sync both the data and the actions taken to produce the data (event sourcing) and try to eliminate sources of indeterminism.

why both? isn't that wasteful? which one wins if they disagree? do you sync both at the same frequency?
It depends on the business logic. If e.g. a user updates a numeric value on an entity, and there's a conflict probably you choose the most recent value. The action taken (user updates value) is pretty useless in this case. In other cases, e.g. inventory reduced by X because X units purchased/consumed - then it's more important to have the action itself, and a conflict would involve applying both actions and possibly alerting someone if that causes the value to be invalid (e.g. inventory below 0.)
Except for the part where even if you ignore the sales and marketing expenses they're barely breaking even. They're failing to make software for less than they can sell it for, and perhaps some of those hard problems which took 50 engineers to solve would have been better to have gone unsolved even if they lost some sales as a result.
That's normal in a SaaS. They gear everything towards growth in revenue, not necessarily profit, since it's much easier to keep existing SaaS customers (most good SaaS have 95+% retention) than to earn new ones, and at a certain point you've got a flywheel going so that you can perhaps reduce headcount increases or stay flat while your revenue continues to grow, and then you coast into profitability from there.
and a team told their manager they needed to redo all the unit tests in order to stay relevant
I mean, I think the real answer is that most companies use inexpressive languages with heavy emphasis on mutation.