[montagg]

I work at a company that grew from a few hundred to a few thousand people in a few years' time, and there was an identical mindset and identical problems over that growth period. I started during that growth period, and 10 years later, we're still digging ourselves out of those mistakes.

One reason for this was how heavily we prioritized first mover advantage. We pushed extremely hard to get into new markets, and while strategically this worked out extremely well for us, it meant that later development was hamstrung by earlier decisions. Any iteration on old features almost always meant rebuilding it entirely, and we've only just started to get to a place where previous efforts to do this are now paying off in increased velocity.

I think there's an argument that this could be good, since you're only improving features when there's some other value you can ship. However, I think it's short-sighted, because it means prioritizing small iterations has an unnecessarily large cost, simply because that cost wasn't paid down slowly over time. While I do think prioritizing new markets early on was a good call, that mindset bit us in the ass in a big way when we tried to ship a very flashy new feature later in our growth cycle that crashed and burned because of tech debt, and it took us over a year to fix everything to release it. If we had taken a mindset of building for the long term five years ago, which was already years after we'd gotten to a comfortable place in revenue and marketshare, we would probably have paid off most of the major tech debt by now.

I honestly don't know how a company can get past the addiction to shipping, at least not during its growth period. It can be strategically necessary, and I imagine in only a two-year period at the early stages of a company, like what's in this article, it's impossible to avoid. But leaders must have in the back of their heads the idea that they will need to start tipping the scales toward standardization, building to last, and building infrastructure as well as features sooner than they'd probably like to avoid larger problems in the future. And I suspect the only way leaders will really take that seriously is if they've gone through it before.

EDIT: Clarified a few points.

[jamil7]

Apart from the one catastrophic failure you mention it sounds like, from a business perspective this worked? You were able to extend your tech debt long enough to start generating money and are now in a position to pay it back. From an engineering perspective I agree it's a grind to pay all this back and lose so much velocity. Are their business strategies that favour slower more careful engineering practises? mission-critical systems come to mind but maybe also nth-movers could be put in there? Where you know what you're building and competing with but want to make the best version of X.

[montagg]

Well, yes, I am saying from a business perspective this worked. but I also think we got lucky. Around five years ago, a little before the high profile failure, we were starting to lose marketshare, and though we've been able to claw most of it back, I think it's far more due to missteps of our competitors than our own performance. Had we invested in more sustainable development earlier, I think we would have been able to move faster to take advantage of new opportunities, diversify our revenue streams earlier (adding to them now is hamstrung in large part because of tech debt), and head off competitors faster.

In fact, once we started investing in that kind of development, we did do that with a successful new feature that we developed much more quickly than we could have in the past, and it's because of that success that leaders are finally starting to prioritize building standardized and reusable infrastructure.

EDIT: I think another way to say this is, once we knew our strengths and how we'd be successful in the future, we should have operationalized that and hardened it, so we'd have a sustainable advantage over competitors. Instead, we kept the "move fast, be flexible, and throw man-hours at problems" mentality too long, and it put us at risk.

[OliverGilan]

Could there be a hybrid approach? Have teams that focus on being a first mover and shipping and then once there's a product on the market, you have a secondary team focusing on fixing the tech debt early while the other team continues to iterate

[wastedhours]

At my old company we had an innovation consultant come in and tell us that kind of approach was nearly impossible to manage in the longer term. Apparently teams dedicated to holding the fort grew bitter and had much lower retention than teams on greenfield work.

Paying back the debt, he said, usually requires too much knowledge of a system to cycle teams in and out consistently enough to ensure everyone gets a piece of the new versus the old to keep them engaged.

Whether or not that was backed up by hard data, I'm not sure. It was an interesting angle I hadn't previously thought of though.

[spockz]

Calling it “just holding the fort” would indeed be disadvantageous. If instead the teams would be empowered and rewarded for also keeping things running and improving on them it will attract the right kind of people which do grow the product and don’t get bitter. However, I have yet to meet an engineer who is comfortable maintaining a sunset product and in the mean time getting only flak about the performance of the old sunset application/feature or their own performance.

[dane-pgp]

> Apparently teams dedicated to holding the fort grew bitter and had much lower retention

As someone who found themself in the "holding the fort" team, I can confirm that this analysis is at least anecdotally true, although the terms I preferred at the time for the two teams were Morlocks and Eloi.

[spaetzleesser]

“Apparently teams dedicated to holding the fort grew bitter and had much lower retention than teams on greenfield work.“

Very true. The good performers will be “rewarded” doing the things that are really important: keeping things going. And then you have low performers and interns doing the new stuff.

[nitrogen]

That can happen too, though I think you're saying the opposite of the parent comment.

[r0ze-at-hn]

Worse than the fact that those holding the fort would leave, the best of those would leave (either to another company or just as often to the exciting new projects) leaving behind those that can't leave and very rapidly you end up with a not that great group which management doesn't notice or deal with.

[montagg]

Early on, I don't think so. You need all hands on deck to move fast. But I do think you need to start establishing those Good Habits way earlier than they might be immediately valuable. For example, getting good at product and technical documentation before you have a huge company. It's so much harder to establish those habits with more people, so you want to have that built in so new folks join, and doing that stuff right is just part of the culture. A little effort today saves a ton of effort to establish those habits later. You also end up with more flexibility: Person A doesn't need to constantly support Feature A because they built it and they're the only one who knows how it works, so you're getting more value out of their time.

In the long-term, I do think product-wide you always have teams at different levels of maturity. Some know more about the problems they're solving and how best to solve them, others know less. Some will move fast, some will build to last. The more a team knows about their problem, or the more a problem cuts across teams, the more I think they should build to last, so investment in those areas scales over a large group or important areas of the business.

[coredog64]

Other people are also commenting, but IME, completely disassociating “making the think from “making the thing so that it can be supported” is a recipe for trouble.

There’s not an easy fix though, as most of the time I see orgs just punt at some point and allow the dev shop to write off all their tech debt by reassigning responsibility to an ops team.

[sundbry]

Then you just end up with the first team continuing to produce garbage, and leaving it up to the 'fixers' to clean it up.

[spaetzleesser]

And the innovation team will have stellar resumes while the people who fix things will look like outdated dinosaurs.

[JackFr]

That is an instant morale problem.

[RandoHolmes]

> Had we invested in more sustainable development earlier, I think we would have been able to move faster to take advantage of new opportunities, diversify our revenue streams earlier (adding to them now is hamstrung in large part because of tech debt), and head off competitors faster.

While that's probably true, it's a difficult balance to know when exactly you should start moving back towards the better development practices. Too soon and you die, too late and you can no longer move because you're too inefficient.

[codemac]

> Are their business strategies that favour slower more careful engineering practises?

This is a false dichotomy, and there is plenty of research (and experience) that shows shipping higher quality software actually takes less time and money.

[jamil7]

> This is a false dichotomy, and there is plenty of research (and experience) that shows shipping higher quality software actually takes less time and money.

I believe you when the business knows exactly what it's building. In the context of the article and the comment I was responding to, building high quality software wouldn't have helped them early on, thats kinda the whole point of tech debt and the discussion here. Knowing when is the right time to start changing the engineering culture and how easy (or hard) it is to do that.

[LanguageGamer]

I want to believe this. Do you have any citations?

My experience would suggest shipping higher quality software is a lot cheaper in the long run, but has up-front costs that often times startups can't afford.

[sundbry]

You can afford them if you don't have project managers or sales guys begging for their demoable results of new shiny thing and promising it to others too early and too often.

[nicoburns]

My companies first 5 hires were junior developers. This cost a fair packets (because there were 5 of them!) and led to pretty low quality software (because they were all junior and didn't know what they were doing).

They could have shipped higher quality software quicker by hiring 2 senior developers for similar cost.

[nitrogen]

Startups bias toward hiring some less experienced developers by their nature, so they don't always know how to build quickly and solidly. The up front cost is either learning skills of abstraction from scratch, or hiring people who have.

[Spivak]

These points aren't as opposed as you're making them seem. Throughput is better with careful planning and good design is an investment that pays off later.

Slapping it together pays off today but at the cost of acquiring a debt that costs lower average velocity and lower agility until its repaid.

A lot of small businesses make the decision to take on debt to move quickly at the beginning because the most important thing is initial growth and proving that you can actually make money.

[mlyle]

I think that they're saying, in retrospect, they could have started making some payments on the debt a bit earlier when the business had scale and sustainability. And that in turn this would have avoided a lot of unpleasantness.

It's hard to change culture; you tend to keep doing what worked before even if it's not optimal anymore. Yesterday's adaptive behavior is today's unfortunate practice.

[jamil7]

Yeah OK that makes sense.

[macjohnmcc]

I worked at a small company that was always chasing the next potential sale. Potential customer mentions feature A and all of a sudden all development shifts towards developing that feature. The vast majority of the time the potential customer never became a customer. In the meantime competition focused on their own plan of features and our product wound up with features no one ever wanted and was missing critical functionality everybody demanded.

[drspacemonkey]

I've had the exact same experience. Throw in a large dose of "no repeat customers" due to us prioritizing Feature A for Potential Customer B over fixing existing gaps that affected actual customers, and it was a never-ending parade of haphazardly created, bug-riddled new features that never got improved.

Made every single day a death march and a company that survives purely on the sunk cost fallacy on the part of the investors. They've already dropped $MILLIONS into building this thing, and the CEO swears that this next big deal will be what finally pushes the company into the stratosphere. As miserable as it was to work there, I'm honestly impressed they're able to keep that zombie moving.

[ryandrake]

Yup, and the requesting "customer" always convinces your CEO with the same argument: "Just add this feature, sell it to us, then it's part of your product forever and you can sell it to many other customers! We're paying for your NRE!" and CEO falls for it!

[macjohnmcc]

The company I worked for didn't even need that incentive just the potential to make a sale! We added features just to complete bullet points on a marketing sheet even if we never made a sale of the feature.

[jorblumesea]

The core issue, from what I've seen, is one of incentives. Many leaders at many companies are compensated on growth and short term factors. It's hard to find a leader that is willing to take a 5 year look, or even find a leader that is planning on being around 5+ years.

The leadership at my publicly traded company is a revolving door of short term hacks for promos followed by a quick pivot to the competition or another company. It often feels like the C-Suite at most companies thinks at most, 2 years out.

I just can't see many leaders thinking 5 years out.

[coredog64]

I would add that when it’s not a revolving door it’s frequently someone who used to be technical and still fancies themselves as an elite coder. That in turn leads to low level decisions (how and not what) being made at the top. As a result, you’re chasing “cool” rather than “best long term decision” in service of the ego of someone who is already being appropriately compensated monetarily.

[satyrnein]

My previous employer prioritized shipping, won the market while having both more features and more technical debt than competitors, bought those competitors out as they started failing, and then migrated their clients over and threw their beautiful (but less featured) systems right into the garbage. To this day, I sometimes think back to how a particular competitor had the right schema for solving current problems, but they didn't even survive to the current day, which takes precedence.

[RandoHolmes]

> One reason for this was how heavily we prioritized first mover advantage. We pushed extremely hard to get into new markets, and while strategically this worked out extremely well for us, it meant that later development was hamstrung by earlier decisions.

I've seen so many developers over the years who view having to rewrite that module because it can't scale well enough anymore as a failure, but I call it a success. Software is meant to be tumultuous, things SHOULD be getting replaced.

In another post in this thread I mentioned that software systems are like children in that they have phases. Assuming that your company and software team is dedicated to fixing those issues (slowly, beside new features), then I think you're probably in a healthy spot despite the pain.

An observation is that business interests must ALWAYS trump technical interests when those business interests are critical. No business = no technical. But at some point when the company becomes stable that balance has to start shifting back towards the technical or the company itself becomes ridiculously inefficient.