While the article delivers nice explanations for what kind of technical dept exist, it stays very generic and hand-wavy for the solutions. Bringing "microservices" into the room is definitely not helpful. Ownership and setting a quality bar..., sure but how does that look like? As I developer I maybe have an idea but I also need to sell that to management. As manager I have an idea too but I need to convince my developers to strive for quality and sustainability.
Sometimes I wish these articles would just state: Stop sprinting
Also: document your decisions! If you have to write down and justify your (prudent) technical dept, you maybe catch some stuff before it's written in code. It also makes sure more people are aware of it. Additionally, it sometimes prevents CV driven development, which imho is a big problem in tech. Overall, documentation is a very underused tool.
Finally: teach everyone (especially marketing) that software development is expensive, so everybody thinks twice before wishing for nice to have features done quick.
Could not agree more. The solution to tech debt is the one nobody wants to hear: Slow down!
Steve Jobs' presentation of Snow Leopard should be mandatory viewing for every manager. "No new features" to a standing ovation. Customers actually like stable, responsive, and efficient tools, who would have thought?
EDIT:
> Bringing "microservices" into the room is definitely not helpful
It's a pretty good way to increase technical debt. The microservices themselves might become small enough that it is easy to refactor some at a time, but then you have the actual deployment and communication infrastructure to contend with. And if you get that architecture wrong, oh boy you'll be begging to go back to a monolith in a day.
> It's a pretty good way to increase technical debt. The microservices themselves might become small enough that it is easy to refactor some at a time, but then you have the actual deployment and communication infrastructure to contend with. And if you get that architecture wrong, oh boy you'll be begging to go back to a monolith in a day.
I remember at a previous job, the monolith was drowning in technical debt. Someone decided the solution was Go microservices.
Fast-forward 18 months and 95% of the functionality is still in the monolith, but there are now 25 microservices, and no environment (except production) where you can test everything together.
> I remember at a previous job, the monolith was drowning in technical debt. Someone decided the solution was Go microservices. Fast-forward 18 months and 95% of the functionality is still in the monolith, but there are now 25 microservices, and no environment (except production) where you can test everything together.
I have the opposite anecdote. Currently working on a monolith that's been around for 5-7 years, huge enterprise Java mess, like many others in the industry. The clients decided that they'd like to upgrade to JDK 11, newer frameworks, all of that shiny stuff.
So for the past 4 months i've essentially been pulling my hair out and trying to rewrite significant parts of it all. When you have a monolith, you cannot upgrade the entire system if some parts of it break - even if i have, say, 200 dependencies but 10 break, i cannot move forwards with the updates and as a consequence am stuck with running on JDK 8 or even outdated frameworks.
And, of course, you cannot extract those parts of the system out either because you'll immediately be hounded by developers who aren't welcoming of change and will find nitpicky stuff to tear both your arguments and efforts apart, actively sabotaging any potential successful outcome (potentially exaggerating here, but many do not enjoy change).
Alas, there is probably some sweet spot to work towards from day 1. Not going crazy with microservices, especially due to how people interpret the "micro" part (e.g. service per person vs service per team, what the total count should be, domain modeling etc.), but not sticking to a single large monolith either.
I think that sooner or later the industry will try grouping code into services based on the "type" of functionality - the weird PDF export and reporting logic will live in service A, other attachment upload/download logic in service B, the web API in service C, and the old legacy server side rendered UI in service D. That way, at least your efforts to update the web framework and JDK for it wouldn't be usurped by the PDF library not liking it.
Then again, i've seen front end applications baked into back end applications instead of separate back end/front end deployments far too often, so i'm not hopeful about anyone genuinely exploring that approach anytime soon.
I don't disagree - this mononlith was running on three to four year old dependencies, and on the rare occasion that someone did make any big updates it was a case of hoping that the tests passed and then hoping that nothing broke in production.
But it wasn't broken up because it was too hard to update dependencies, it was broken up because it was a complicated mess. They just traded it off for another complicated mess.
Snow Leopard is exactly what I had in mind when writing the comment. I remember watching the keynote and my own excitement about introducing no new features.
Additionally to managers watching that presentation, software architects should mandatory have to work as normal developers in every project they designed for a a few months. Best, if it would be around 2 years after they came up with the architecture. If you don't taste what you cook, how can you learn?
This happens constantly, the entire tech industry runs in whatever direction Google in particular is going. First it was Hadoop and all the pain that came with that, and now it's microservices and kubernetes and all the shit that brings into our tech stacks.
Stop sprinting is good but also : stop following Google. You are not Google. You do not have the needs of Google. Just stop it.
Pick the simplest tech stack you can find. This gives 100x more time to your developers to work on company solutions and not fighting technical complexity.
It never ceases to amaze me how much effort we now spend on nothing but infrastructure and architecture. It's like the Cloud Way of "cattle not pets" and "infinite horizontal scalability" may no longer be questioned.
Meanwhile you could build almost anything in this space with little more than a rack of real servers, some respectable application code written in a fast language, and a bit of scripting and OSS for the glue.
Obviously that sort of environment wouldn't be sufficient for every modern, online-first application but how many years would it take to outgrow it? Many applications never reach that scale. The lucky few would be in a great position to expand from if their starting point was simple and transparent with minimal dependencies.
The problem is you eventually need the things k8s offers and you end up implementing part of k8s, badly. I've seen bigish places that state "we aren't Google" as an argument to not use k8s, containers, etc but don't consider that their tech stacks are a Hodge Podge of bespoke glue, a mass of technical debt and a bus factor of generally 1 or 2.
The question they should be asking isn't "is this tech stack perfect or a mess" it's "is this actively preventing, or about to prevent, us from meeting our uptime and cost and development goals"?
The idea that a shop that built a messy system will suddenly build a non-messy system just if they pick up a new set of tools - that they don't have practice with anyway! - is exactly the silver-bullet fallacy. The system turned out messy because it's complicated, switching to a very complicated tool doesn't take away the complexity, it just changes the way you manage it.
The last couple Kubernetes shops I worked at were that exact sort of hodge podge of custom glue with high tech debt and low bus factor. Just even doing something "simple" like "ssh into the box and poke around" when the two wizards are out suddenly has a huge learning curve. ;) So then you get into discussions of "oh you should've been using managed kube" or "oh, don't use TF -> helm -> kube, use [other way] of managing it" or blah blah.
After you fuck it up a few time you might have enough useful experience so that you can build it well next time... just pray that by then we're still using the same tools. :D
There is a huge learning curve with k8s and there really needs to be a "k8s the good parts" for tooling to use. The problem with k8s vs $custom_system is k8s is somewhat well thought out and $custom_system is generally not.
As someone who works on a hodge podge of bespoke glue, a mass of technical debt, a bus factor of 1, and a docker swarm setup, I don't really see how k8s would help ;)
> Customers actually like stable, responsive, and efficient tools, who would have thought?
_Existing_ customers like these things (and they aren't wrong for liking these things).
New business comes from the potential customers who weren't enticed by the existing feature set, robust or otherwise. You bring them in by adding new features. The retain/new business priority is often heavily weighted in favor of the latter.
Retaining existing customers stuck with a crappy product ain't that hard between sunk cost/lock-in effects and the evergreen insulation of people with purchasing authority from the day-to-day pain inflicted by their purchasing decisions. A couple fancy steak dinners for middle management effectively papers over the cost of driving a department of ops engineers to cirrhosis several times over.
Slowing down is really being able to say "no" to features for awhile while you re-architect the codebase - so you're delivering stuff at the same pace, but that stuff is not product-facing.
I've been through this slog - sometimes it requires fixing what's there and other times, a rewrite. People hate rewrites, but it really is the best way forward in a lot of cases if done correctly.
This is my experience with many different systems. I’ve found that mentoring and personal development identification techniques for what to learn, and when, can be helpful. Inviting a developer to be part of the decisions made regarding their future may seem obvious, but in practice it’s often not done…
And documentation… I’m known as “Just write it down,” but there is usually significant pushback… from all parties.
It's kind of ironic reading that from Martin Fowler that is for me the astronaut architecture goto person :). I have huge respect for this work but I guess a lot of technical debt is probably caused by implementing these complicated pattern - add some AbstractSingletonProxyFactoryBean joke here.
I pretty much lost respect for him when he started advocating short methods.
From [1]:
> In my Ruby code, half of my methods are just one or two lines long. 93% are under 10.
Are there professional engineers out there who reads this tweet and thinks striving for one or two lines methods is a good idea? I really don't understand how does this person have so much fame or so much authority on architectures? What exactly are his accomplishments other than selling books and seminars? Can someone please explain how come Martin Fowler is so revered among developers?
> Are there professional engineers out there who reads this tweet and thinks striving for one or two lines methods is a good idea?
Striving for the shortest possible methods (less than 10 lines is good) is a standard in Smalltalk. It's not a bad idea, at least, it's been shown to work over and over again over the last 50 years. Ruby takes a lot of inspiration from Smalltalk, although it lacks the biggest factor that made short methods work so well in Smalltalk: the interactive editing of the code as it runs.
Another example where a subroutine longer than a few lines is frowned upon is Forth. Also in that case, interactive editing is the main motivation, along with the fact that the amount of irrelevant bookkeeping grows linearly (or worse) with the subroutine length (unless you use local variables or the return stack, which are both hacks in Forth).
In any case, "professional engineers" who don't even know Smalltalk... No, I mean, the history and breadth of their field, probably should not be too judgemental about what works where, outside of their immediate expertise.
1-line methods is taking things to an extreme (though you can write a lot of logic into a single line in Ruby, so maybe it's not as crazy as it would be in C...) but I found the discussion in his refactoring book about reorganizing methods and naming things very illuminating.
It's an idea so simple it sounds stupid sometimes to try to explain to people, and yet, I never stop seeing new code that could be improved by something as simple as methods with accurate names.
E.g. I've seen a lot of stuff like:
```
// This does X
[some obtuse nasty inline regex or five-layers-deep nested object call or otherwise crazy individual line of code]
```
That maybe eventually gets detached from that comment line and now there's just some nasty line that nobody on the team understands anymore that can't as-is be easily tested in isolation from a bunch of other stuff.
A perfect opportunity for a one-line method, that can have a descriptive name and can have its own specific tests.
And yeah, again, that's the extreme case, but most codebases have tons of opportunities for 5-10 line methods being extracted with helpful names. If you find yourself writing a comment to describe a block of code, maybe make that comment your method name instead. And sometimes you try it, and realize "wow, it's hard to extract these methods without an insane amount of input params and return values for each bit" and maybe that's an important thing to realize about the code you're looking at anyway. ;)
Through happenstance, my "first" specialization (subsequent experiences demote it to third place, as skills I discounted became more appreciated later in life) in the industry was performance analysis. Over time I came to understand the overlap between Knuths admonishments on optimization and "too clever by half" coding (ie, the large intersection between Knuth and Kernighan), and rather than abandoning it entirely I channeled that effort into finding more benign forms of optimization. There are code smells that are slow. There is slow code with higher cyclomatic complexity than a faster version. There is slow code that is less maintainable. These are not hard to find if you look for them.
The one that snuck up on me was sort of the corollary to Screechingly Obvious Code, and a cousin to Pandora's Box - which is, strictly speaking, and antipattern (except that in the end of the original story, Hope was left in the box).
If you have a function that is Obvious except for one small part, factor it out into its own function. The care comes in picking the incision points. The inputs and outputs to the new method need to be obvious (ie, don't factor out a function with side effects, that makes for spooky action at a distance), which may require some temporary data structures in order to iron out.
Going further, if you are considering a behavioral change that may not be obvious (performance optimization, error handling for very obscure cases) that has no prayer of being screechingly obvious, determine what lines will need to be changed, extract method on that code first, commit it, and only then make the change to the extracted code. Mikado method often comes into play here. Get used to revisionist commit history techniques.
What I've found through long observation is that people are more comfortable reverting such changes if they break when you are not there, which in turns make them more open to approving the change in the first place. "Will this bite me on the ass later?" is a pretty important consideration for most people. The commit history is very clean, and someone can revisit your decision years from now, when your user base and hardware are orders of magnitude bigger, when the runtime and hardware have very different timing ratios between different activities, and so on. And critically (for the optimization obsessed), compilers are very good at inlining leaf functions, which means the 'cost' of the context switch can be negligible in practice.
Also I don't know about anyone else but I've found that my attachment to such code is also fairly low. You are much more likely to get encouragement to 'go for it' when broaching the subject of undoing one of these changes, instead of a lecture about watching for regressions (I tend to "over-invest" in robustness, so deleting my code statistically represents a reversion to the mean wrt to code quality).
And perhaps more importantly to this conversation, they tend to tune out these leaf node methods when tracing code looking for bugs or trying to add new features, bugfixes. These structures don't nerdsnipe us, causing us to dump state on short term memory and have to start over. If I had a dollar for every time I said, "Aha! I found the place where we return null! Wait, why was I looking for that?" I could retire. This is, in my mind, the critical quality of Screechingly Obvious Code - in uses up a single slot of short term memory, facilitating code traversal.
Cargo cult programming at it's maximum expression.
M. Fowler, Uncle Bob, Kent Beck, GoF authors...
Any book or post by these people is always brought up as a must read. A must read to understand why you should *not follow cargo cults* and see with your own eyes how toxic and damaging these guys are to software engineering culture.
In one hand, I'd say these guys are absolutely revered among computer science students (their main target to sell their books) and developers who have been stuck in the OOP and design pattern rabbit hole for decades without evolving.
In the other hand, the criticism is more present that ever. It's ironic, because the criticism has even lead Uncle Bob to jump the wagon and start to sell Functional Programming in Clojure as a new super *secret formula* and *best practice for real engineers*.
I fully agree. If I see those names attached to a project I take it with an extremely healthy dose of scepticism (and usually run the other direction).
I'm somewhat glad I made it out the other side; as a babe in arms programmer I was fully taken in by these ideas. It's a great game of fun creating these overly complex, but ultimately arbitrary, sets of rules. They serve no real purpose other than creating and solving mental puzzles. There's a reason Software Architects don't (shouldn't) exist any more.
I haven't written any ruby, but I've seen very short methods used in Erlang and it can be useful. If you write one or two line methods, they're mostly to help name a concept and make the place they're called look more sensible.
Organizing all those small functions can be a challenge, though. Which is why I don't tend to have functions quite that small. But a lot of 10ish line functions tends to work ok. But then, Erlang doesn't have loops, and I don't like anonymous functions (unless they're so short and simple I don't expect to find them in a stack trace), so that tends to make things short by necessity.
Techical debt is primarily about managing complexity, the bane of software engineering. Do not underestimate the value of having a coherent _theory_ about the problem being solved. Something everyone on the team understands and that can be taught to new hires. If you give a programmer a keyboard and a paycheck they will press on keys all day long.
And if complexity can be designed out from day zero with a clear and concise foundational development model to follow then the technical debt can be greatly reduced as well. This of course conflicts with the global rush into tech to build something as fast as possible to get to revenue and not investing the time proactivity to build for the long game. Planning and control are my two points that get extreme focus in designing that foundation as the world comes to learn the importance of these items impacting cybersecurity, fairly important now-a-days it seems, and so much more. If you know something works then no need to reinvent it every time.
Designed out on day zero is the siren song of waterfall.
Getting it right the first time is a very hard game to win. It’s best to save some of your energy for the times that really count. It’s always interesting to me when coworkers exclaim that doing the right thing is too hard. Reminds me of myself at age nine trying to get out of chores. More seriously though, “if it hurts stop doing it” is how dumb animals think. Pain is information. Ignoring it is dumb. Almost as dumb as giving up is.
Just last week I was having a hell of a time getting some code to work. Running into a wall. Okay fine, I’ll write more tests. Still struggling. Oh hey, you know what would make this way easier? If I rearranged this code in the manner I thought about this morning but decided to not work on until tomorrow.
If eating the code is difficult, there’s a point of very quickly diminishing returns where adding more logic to the tests is making things worse, and you should think about whether The code is too complicated to test. Maybe you need to remove code, instead of adding it twice.
After decades of software development experience inclusive of 24/7 on call support while also being the person of which ALL bucks stopped I personally have lived that "hurt" too many times to count. If time can be invested prior by designing out those problems, looking back now, then this is something only experience could bring since one does not know what they do not know.
"Getting it right the first time is a very hard game to win."
I cannot disagree if it is one's first time yet this is my sixth and I have no losses to date. I have nothing but time and so in that time I will see how this game plays out.
Irreversible decisions are a very small subset of all decisions you will make. If you don't do those at the beginning (or write your code to be agnostic to them, ie, by leaving out feature sets that would surface that decision), then you will be in pain. Experience will tell you which those are, and beginners often have to learn this the hard way. Some never do. Yes let's add localization and security on to the application after it's already 100kLOC and shipped to paying customers. That'll be fun.
Everything else is just annoyance, and lumping those in with the others ends up pulling attention away from the ones that really matter, meaning those solutions get compromised on quality.
Leaving space for a solution is not the same thing as creating an abstraction for a solution, and most people don't get that. Architectural astronauts elevate this blind spot to performance art, and none more profoundly than J2EE architects, as lampooned by Enterprise FizzBuzz: https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpris...
I wonder sometimes if some of my former coworkers' cognitive dissonance ends at the office door, or if at home they hang empty picture frames on their walls as placeholders for when they eventually find some art they like, put armchair-shaped boxes labeled "Chair" in their living rooms, dolls in empty cribs and stuffed animals in empty pet beds.
Technical debt is about managing accidental complexity. But well-placed accidental complexity can make inherent complexity look much, much bigger than it really is.
Clearing debt can be a bit of a fishing expedition, but as long as enough of them have a payoff, it hardly matters if occasionally they come up goose eggs.
Over time, I've come to believe discussions around "Technical Debt" do not sufficiently examine the nature of the risk of the underlying challenge. The framing also skews and pigeonholes the responsibility part of addressing things.
In short, I feel a re-scoping in terms of "Software Debt" is warranted. And a re-casting of the risk of this debt in terms of the rocket equation, and opaque financial derivatives of the kind made infamous in 2007/08.
It can happen. Sometimes it even happens on big projects that are deep in debt.
But it only ever happens when the hands-on people get to spend significant time on improving what is happening behind the scenes. That might mean 20% or 50% of their time, not the 2% or 5% that many get.
If the hands-off people calling the shots aren't willing to make that commitment then a heavily indebted organisation's fate is already sealed.
How we talk about tech debt, how we argue about its importance and how different functions in a team (tech and non-tech) can get to a common ground is usually the most important and many times the most overlooked part of this. Teams tend to fluctuate between "everything is important" and "our PM never lets us do any tech debt relief work".
There’s sometimes bad code that one resolves by redesigning some part of it with eventually some glue code with the old system, documenting it and making a presentation for everyone on how should the new stuff be written, and distribute work for a slow migration of the old systems by the different parties.
I’ve seen also some systems designed to solve problem X while the company or industry moved on to problem Y. Again, act like a virus, infect the system with your brand new solution, glue it or bind somehow with the old system and let it spread.
The only mistake that you can do is to engage all of your resources to undertake a full redesign and rewrite everything from scratch.
And for processes or algorithms that don’t scale, it’s not a debt because you never really made the investment.
I am starting to believe that software has a shelf life. Some of its fresh fruit and milk and some of it is canned beans that will survive for a long time. An answer to technical debt is throw out software
Sometimes I wish these articles would just state: Stop sprinting
Also: document your decisions! If you have to write down and justify your (prudent) technical dept, you maybe catch some stuff before it's written in code. It also makes sure more people are aware of it. Additionally, it sometimes prevents CV driven development, which imho is a big problem in tech. Overall, documentation is a very underused tool.
Finally: teach everyone (especially marketing) that software development is expensive, so everybody thinks twice before wishing for nice to have features done quick.