[kqr]

Something I've recently realised after having listened to Kevlin Henney talk about software engineering is how much of the existing knowledge we ignore. The early software engineers in the '60s and '70s were discovering pattern after pattern of useful design activities to make software more reliable and modular. Some of this work is really rigorous and well-reasoned.

This is knowledge most engineers I've met completely ignore in favour of the superstitions, personal opinions, and catchy slogans that came out of the '90s and '00s. It's common to dismiss the early software engineering approaches with "waterfall does not work" – as if the people in the '60s didn't already know that?! Rest assured, the published software engineers of the '60s were as strong proponents of agile as anyone is today.

Read this early stuff.

Read the reports on the NATO software engineering conferences.

Read the papers by David Parnas on software modularity and designing for extension and contraction.

Read more written by Ward Cunningham, Alan Perlis, Edsger Dijkstra, Douglas McIlroy, Brian Randell, Peter Naur.

To some extent, we already know how to write software well. There's just nobody teaching this knowledge – you have to seek it yourself.

[TravHatesMe]

Agreed. But something I'd like to add:

You can read all the papers you want. At the end of the day, you actually need to practise writing code! Every piece of software involves a different domain, different requirements, etc.

I feel like the majority of developers these days just copy and paste stuff from the internet and glue some npm/nuget/maven packages together, brush their hands together and feel like gods. That is not how you become a good developer!

How do you become a good developer? Write code _yourself_. Then rewrite it again and again. Keep thinking "how can this be done better, cleaner, more elegantly? how could this be more readable, maintainable?" .. "Maybe others have found a better solution for this, let's do some research and investigate all options, weigh the pros and cons, and select the solution that fits best for this particular problem". Don't forget to factor in the cost of complexity. Is the code you are writing too difficult to understand for the next developer? Is the abstraction too rigid and complex? There is a cost to that.

Rinse and repeat. Sooner or later, once you've written enough code and thought critically about every line, you will acquire a "feel" for what it right and what is wrong. Then suddenly you are an industry expert. Suddenly you are training and mentoring others. Because you put in the goddamn effort.

[collyw]

The thing that improved my coding was actually maintaining my own codebase for a few years (4.5 years in the same job working on a system that I wrote from scratch). That way you see which of your decisions worked and which ones were crap. When you need to revisit your own code 6 months later and can't understand it, that's no one else's fault but your own.

Even then I think you have to be able to look at your own code critically and want to make it better. The current tech lead in my company churns out a lot of code quickly, but doesn't take constructive criticism of it at all well. It's not easy code for other people to jump into and there are no docs or tests. He thinks it's close to perfect. I don't

[cmwright]

This is an issue I've seen with programmers that hop around companies every year or two - they rarely get to see how their software choices play out in the long term and so never really internalize what works and what doesn't. Same issue to some extent with people who keep hopping around languages/frameworks - if you only have experience writing projects from scratch you'll never really understand what works and what doesn't from a long term maintenance perspective.

[kqr]

This is also why I have doubts about "experienced consultants". It usually means "job-hopping far more frequently than every year."

[mannykannot]

Part of the thinking about it is not making excuses to yourself. If I have to rewtite something, I don't just dismiss it as refactoring, I also ask myself if I could have got it right the first time. And try to estimate how long it will take, because you will improve with practice, and it is part of doing a professional job, even if it is thoroughly broken and abused at your place of work.

[collyw]

Agreed. I think maintaining your own code for some length of time is at least as important as writing new code.

[pteraspidomorph]

> feel like gods

I've been disappointed with the industry for the opposite reason: I don't want to be doing this, but this is what is being asked of me.

[whytheplatypus]

Couldn't agree with this more.

A current favorite of mine is "The Emperor's old cloths", the Turing award lecture given by C.A.R Hoare. In particular his line "The price of reliability is the pursuit of the utmost simplicity", it applies equally to maintainability and extensibility (I think these are part of what it means for a system to be reliable).

An idea I try to keep in mind while working is not to plan or build for future features but simply leave room for them (meaning don't actively prevent their eventual existence through complexity). It has taken some practice, but it helps guide to a simpler implementation.

[kqr]

> maintainability and extensibility (I think these are part of what it means for a system to be reliable).

Frequently forgotten is the duality of extensibility: subsettability, or contraction. Being able to remove or disable code without rewriting large parts of the application is just as important as being able to extend it!

[whytheplatypus]

yes!

[CuriousSkeptic]

This. So much this!

Especially Microsoft has a tendency to throw out frameworks and helpers that just don’t provide the needed flexibility, and once that has been addressed what’s left is a heap of extensibility points that wrap three lines of useful logic in hundreds of lines of framework code.

So if I could ask one thing of platform developers it is to leave the framework design to the user, and just provide useful primitives. The two hours spent on writing the extra glue code is easily saved on the days spent trying to learn the framework.

[teknopaul]

Thats a gem: "is there room for features" is going on my personal code review checklist. Right after "you arn't going to need it".

[ngneer]

And security!

[mannykannot]

One more unsung hero: Barry Boehm, who, in "A Spiral Model of Software Development and Enhancement" (1986), expounded on a iterative process in which each cycle is driven by an asssessment of the biggest risks threatening the satisfaction of the stakeholders' 'win conditions'. Does that sound familiar?

In answering the original question, one would hope that focusing on the risks to successful completion should de-prioritize the fixing of things that are unlikely to become problems in practice.

https://dl.acm.org/doi/10.1145/12944.12948

[andi999]

Well, in some domains there are big changes, due to hardware working differently. When doing number crunching nowadays cache locality is king (cache is 100 times faster then ram). So one should start from the memory layout of the data (data driven design?).

It even lead to Stroustroup saying in a talk that one shouldnt use linked list at all, but instead vectors, because even when linked lists should shine, they do not anymore.

[SamuelAdams]

Yes exactly. I recommend reading the book Design Patterns [1] for some ideas on how early "modular" software was conceptualized. This book was published in 1994 and still has a lot of relevance today.

[1]: https://en.wikipedia.org/wiki/Design_Patterns

[collyw]

One thing I notice is that software is becoming more brute force and less about "design" or "architecture" quality.

Good abstractions at the correct level and the correct data modelling can make the code around it very simple. Instead of nice designs that are easy to reason about, we have lots of unit tests to "prove" that it works. Instead of making a system reliable and trap errors properly, we use Kubernetes to launch another pods as soon as one goes down. Maybe I am just too old, but I feel we are loosing something going down this route.

(Ok, there is code that's functionality will mean that in inherently isn't simple, but I see far more CRUD apps that are built in an over complicatd manner, than code that is actually doing something algorithmically complex).

[kevstev]

This is about cost optimization to some extent.

Writing perfect bug-free code is nearly impossible and expensive. Hardware is going to fail, network links can get flakey, packets can get lost. Accepting that failure is going to happen no matter what and building for resiliency to it is a far better approach.

Unit tests are along the same line- its been measured that fixing a bug in production is 10x more expensive than finding it in development. Unit tests help ensure that more bugs are found in development, there is less need for, slow, expensive, and error-prone manual QA processes that hold up release frequency.

Unit tests also help document the code to some extent. As teams turnover and new developers who may not fully understand the nuances of some code come in, Unit tests create guard rails that help catch edge cases that may not be obvious to new members of the team and when the test fails makes you go back and give more thought to those areas and why they behave the way they do.

Rereading this, you claim to be on the older side, which I am surprised about- this seems to me like an opinion that someone who is relatively new to the industry may have. I am continually amazed in the new and creative ways things can fail, building for resilience is a 100x better approach that also reduces stress tremendously- Instead of "All hands on deck no one is going home tonight- we just took the system down with this last release and will need to patch it ASAP" the conversation goes to "Ok- we rolled out the release to 1% of our users and we see error rates spiking. Lets roll it back and look at the logs to see what happened."

Similarly with unit tests- it used to be monthly or weekly release cycles, now we are down to multiple times a day because I don't need a programmer who washed out to get around to clicking on the right buttons to give it a green light.

These are tremendous improvements in the state of the art.

[collyw]

I am not arguing that unit tests are not valuable for the reasons that you have given, just that they seem to be used in place of decent abstractions these days. I have worked on well abstracted code without unit test and badly abstracted code with unit tests. The former is far easier to work with and bugs take a lot less effort to fix - that translates to costs in developer time.

> This is about cost optimization to some extent.

Is all the complexity and developer time that is required with Kubernetes really cost effective? We added it at my work and it took around 6 months of developer time from what I could tell (it wasn't me who set it up). Now we have to debug it and none of our team is an expert, so lots of guesswork and googling. Using number of small pods has given us a lot more problems than one big machine. If you are running out of memory because of a large upload, restarting a pod when it crashes doesn't help you.

Kubernetes has lowered our cloud costs and given us high availability, but it has increased development time in a number of ways. Maybe as we get more experience it will get better, but so far I would not say costs have been optimized as a result.

[wronghorse]

This is certainly an interesting thought. Ivan Zhao of Notion had some similar thoughts on design. That the early pioneers of computing were able to develop a large amount of amazing insights not just due to the greenfield nature but also because they were less focused on the business aspects.

[kqr]

I don't think they were any less focused on business aspects -- other than reliability, most papers and articles seem to focus on bringing costs down.

If anything, I think the Greenfield nature may have more to do with it. Today, we take it for granted that software is really expensive to make, to the point where we don't always question the basic patterns that make it so expensive. In those early days, it was far from obvious that software would be expensive.

[bobmaxup]

We read some of these papers in school as assignments and there are many cited in software engineering texts like Sommerville and Pressman, so I am not sure no one is teaching it.

[kqr]

This might vary between schools, then. My education was heavily focused on writing algorithmically efficient code that was going to be thrown away weeks later. I don't think we ever deliberately practised modularisation, code reuse, changing requirements, simplicity of implementation, message passing for concurrency, and so on.