[Jupe]

So glad to see this stated publicly, and actually getting some positive responses. 25+ years of software development and the best "pattern" I can identify after dozens of projects is just one: simplicity.

If you can't follow the logic by just reading the code, then maintenance will be slow and cumbersome. New development will eventually taper off as more and more time is spent fixing issues. I've written production systems, still in operation today decades later, that hold true to the simplicity principle. I wouldn't do it any other way. But, the battle I frequently face is with upstart devs who want to use some new pattern or principle because they read that it was "better" than straight-up code. And yes, they can implement some new feature quick, but 2 years down the road all hell breaks loose.

Some of my pet-peeve anti-patterns:

* "Plugin" systems only used by the primary development team. There's power in being able to follow the logic for any call stack by just reading the code; plugins make the flow much more challenging, and provide little value.

* Large (and complicated) run-time configuration IOC systems which make any debug session a dreadful process.

* Rules Engines. 'nuff said.

* Piles of abstractions, one atop the other, because teams had no time to find commonalities and/or sensible use-cases.

* 25+ new source code files to respond to a /heartbeat endpoint.

* Fixing a problem in 3 lines of code is not fixing anything if it complicates the debug process, forces a re-write of the unit test drivers, enables mysterious build failures or taxes the DevOps team.

And some of my favorite simplifying patterns:

* AOP for logging; but NOT (dear god) for business logic.

* 3 levels of abstraction, and no more. If you can't figure out a way to do it in 3 levels of abstraction, then something else is wrong.

* 3 systems of state: config (global), current request and persistence (database). If you need more than that, then something is wrong with the architecture. (Ok, compilers may not be able to get away with this, but most applications can, in my experience.)

* If you use a pattern (MVC, for example) apply it to the whole system. E.g. doing IOC only in the upstream API system when the rest of the code doesn't, just makes a mess.

[kstenerud]

SOLID is useful as a rule-of-thumb, but there's always one over-arching principle that should trump all others: YAGNI.

It's easy to build something simple (and perhaps naive) and make it more complex over time as your needs become selectively more complicated and your grasp of the domain more nuanced, but it's impossible to start complex and pare down to a more simple implementation. Even worse, your bad design decisions due to your less-than-complete grasp of the domain become impossible to remove due to all the layers.

There is no One Right Answer (tm) to system architecture; it comes with experience. All of these principles should be considered guidelines to help in your design, not ironclad rules.

[marcosdumay]

Except when you will surely need it, just not now, and will have to change your entire core everything to bolt it on.

People saying YAGNI usually make much worse atrocities than the ones aiming for SOLID. And on SOLID, the I alone may be responsible for the overwhelming majority of the problems.

[hinkley]

Anyone writing a library should be forced to single step through it in a debugger in front of a group of people and listen to all the grumbling.

The thing is if you have twenty devs all making their code as absolutely simple as possible, the overall system is still gonna be pretty complicated. That’ll be challenging enough. Don’t add your own accidental complexity on. Even if you can fit it in your brain, you’ll be “misunderstood” and “have to” fix things for others all the time instead of working on what you want to work on.

[jackcviers3]

Devil's advocate opinion.

Some might say that the library is poor if you have to debug it at all.

Any library you have to debug to make work is, by definition, a leaky abstraction.

Now, if you followed SOLID principles, you could simply swap the library with the leaky abstraction out with another under a new concrete implementation, verify in tests that you are calling the library with the correct arguments, and your problem should be fixed by only touching the wiring code to point to your new implementation. If it is, then you can deprecate the old implementation class, find all the uses via compiler warnings, and replace them with your new implementation. Then all your tests should pass, and the issue should be fixed.

Accidental complexity arises when you can't do this, and it comes from not properly abstracting away the use of others' code from your own and by overly applying both the NIH and YAGINI principles on both ends of the spectrum.

Simple means "of fewer parts." In maintenance, this means you change fewer parts of your system to make a change. In design, this means you use fewer abstractions. The balance is struck when you have just enough abstraction that maintenance and design are simple.

[hinkley]

All software has bugs.

All abstractions leak.

Composition always leads to new problems.

And we are talking here specifically about people using “good practices” in a way that makes the code far more complicated than it needs to be.

The guy I know right now who writes the most tortured code is so stuck on code deduplication that he’s made a mess. Stepping through his code is a recursive nightmare of wiggle words (compounded by his raging overconfidence in his grasp of English). His code has no shape and he likes it that way.

Somebody hurt him and we are all paying for it.

[arvinsim]

> So glad to see this stated publicly, and actually getting some positive responses. 25+ years of software development and the best "pattern" I can identify after dozens of projects is just one: simplicity.

Yup. So much so that if it takes me several reads to understand code, it's a code smell for me.

[hinkley]

“Just because it works, doesn’t mean it isn’t broken.”

[barking]

Sickle cell trait works to protect from malaria.

It works but is broken.

[collyw]

> I can identify after dozens of projects is just one: simplicity.

Isn't that the point in SOLID though? To try and have some guiding principles for achieving simplicity. Certainly I have noticed that my own code hasn't been easy to understand a few months after writing it, one of the reasons used to be that I was trying to do too many things in one function / method.

Of course having said that, everything is usually a trade off and overemphasizing things will probably be counterproductive.

The other problem with ideas like SOLID, is that they tend to be so abstract that its difficult for them to be meaningful until you have enough experience to not really need them as guidelines.

[JaumeGreen]

> Isn't that the point in SOLID though? To try and have some guiding principles for achieving simplicity.

Probalby yes, but some things are an overkill.

"Single responsibility principle" is great, and should be practised.

"Open–closed principle" is also good. No problem here.

"Liskov substitution principle" sometimes feels like it should be more a language feature than an implementation one.

"Interface segregation principle" can be ok. The problem lays on when one creates interfaces for things that don't need one, because they are one of a kind. The code becomes confusing without any real gain.

"Dependency inversion principle" depends a bit on the previous point, because the links are through interfaces.

I've seen (and been charged to maintain) programs with too much ID (of SOLID) done, and it's chaotic.

It makes sense to do them in an enterprise size program, it makes sense to do them as a refactor when the program grows. But people applying principles willy-nilly on programs that do not need them (yet) make it too confusing for the ones after.

[verinus]

This so much!

simplicity is underrated and unfortunately esp. young devs are proud when they are capable to build complex systems.

After working extensively with plugin architectures like OSGi an learning about functional programming I am deeply skeptical when somebody mentions plugin systems and IoC containers.

[0x445442]

I've just recently started using Java 8 on the job and what's really jumped out at me is the slight of hand of the Function interface. By simply creating a generic interface called Function and "tricking" developers into coding to an interface things like IOC containers and mocking libraries are no longer needed.

I don't need Spring to auto wire in dependencies in order to simplify testing. In my test class I can just whip up whatever dummy implementation I want and create add a constructor for the Unit Under Test to inject that implementation manually.

The other thing that's been rolling around in my mind using Java 8 with the Function interface is; if you want to expose architecture issues and other warts in the code base, disallow the use of mocking test frameworks.

[_57jb]

Had a knee jerk response to this around SOLID really advocating for simplicity in a somewhat rigorous definition.

Completely agree with your statements, a good set of guidelines.

A challenge we face is that there will always be those who either don't know what they don't know so will continue to reinvent the wheel, or those that don't feel at ease unless they are doing something clever and difficult to maintain.

I have seen many talented folks from both sides but they are ultimately a cancer to any product.

Forcing them to maintain their mess has helped but most times it is pawned off.

Transparency, traceability, ownership...that I believe is a path of least resistance that could improve things and lead us towards simplicity.

[jlj]

Rules engines come up frequently in my domain area. What are some good resources for managing either high volume or high complexity business rules?

Usually these start out as hard-coding, then evolve into a rules engine or framework. Sometimes after the original devs leave, the rules engine turns into a shiny black box that remaining devs are afraid to touch.

What are some anti-anti-patterns?

[closeparen]

Be deliberate and skeptical about how and why your rule definition language is a better fit for the problem space than a general purpose programming language. When you find yourself recreating a general purpose programming language, stop. Just drop down to one. Or start with one. A very successful rules engine at my employer is Python minus features, as opposed to the typical "config plus features until it becomes a shitty Python."

Realize that what you are doing is a programming language, and create as much of the infrastructure for programming as you can for rule authors (version control, code review, unit testing, incremental deploy, etc).

[zaphirplane]

What about security and safety. A user provided rule in a DSL has controller access to the rest of the environment. While a user provided script in a language will have a lot of security and safety issues, even if you trust the user there is security in depth and safety of limiting the accidental damage

[gdy]

Not sure about Python, but it is very easy to embed Lua in an app in way that executed scripts have access only to what is deliberately exposed to them.

[adrianN]

Sandboxing the code is a solved problem, is it not? There are a number of websites that run code for you somehow.

[FabHK]

It's a surprisingly tricky problem, btw, at least for some languages. Here's a nice 2014 talk by Jessica McKellar: Building and breaking a Python sandbox that gives insight into some pitfalls. Might be "solved" by now though, don't know.

https://www.youtube.com/watch?v=sL_syMmRkoU

[zaphirplane]

Running stand alone and throw away code in a container, is very different from running a user provided script within your long lived application securely. Think credentials, Db access, file system access, network

But you want to access the DB and write to files and the network just not anywhere, so you have different process and communicate via rpc

[jlj]

I prefer to use SQL because it handles the set definition very cleanly. It is basically the "where" clause of a query. And it's a good middle ground where programmers and business analysts can speak the same language to describe a dataset.

Once the set is defined, the programming language of choice can be used for the action. It could also be SQL, or since we use Spark for a lot of our compute it could be Scala, Python, or Java.

I've been in recent discussions about building a new DSL for this, but I haven't been convinced why we need a new DSL when SQL is already a widely supported DSL.

[hinkley]

Bertrand Meyer suggested you split the code for making a decision and the code for acting on it into separate methods. Happens to work pretty well for writing unit tests, too.

If I don’t edit your function, it’s harder for me to screw it up. You can segregate code without pulling it out into an interpreter.

[chriswarbo]

That sounds very much like functional programming styles, where decisions are made by "pure functions" and actions are taken by "interpreters" or "effect handlers". The "ports and adapters" and "functional core, imperative shell" approaches are similar.

[jlj]

Thanks for the lead, will read up on this.

In my SQL biased mind I think of the decision as a dataset defined by some filters and joins. When new data meets the criteria, it triggers an action for that set.

[p2t2p]

Can I ask a different question?

Very often rule engines are introduced with assumption that "advanced user" with understanding of the business side of the issue would provide (create/modify/remove) set of rules for the system.

More often than not it happens so only devs are able to modify those rules.

So the question is: why is it easier for you to write business logic in some kind of DSL instead of in the actual programming language?

[thrower123]

I have never seen this work out as intended - at best it results in developers having to code up things in a limited way through a leaky abstraction. I've never seen users be actually able to use such systems.

The best that can be said is that it sometimes gives the ability to do some customization or extension without having to do a full deployment, but it also locks you into an ever-expanding surface area of code that needs to be supported and increases the chances of bugs from unforeseen interactions.

[mdpopescu]

Constraints. I prefer not having the freedom to do anything I want in some cases. Pretty much the same rationale as the one behind removing the "goto" instruction.

[samcodes]

One time we put these rules in a graph database. It worked better than hard-coding and a rules engine.

[Jupe]

Ha, funny you should ask... I've started a project about a month ago that will apply various "rules" be applied to unique resource instances. (Sorry, I'm being explicitly vague as there are IP concerns here.)

Here are some approaches that I find myself gravitating towards with this project (but still in design/early dev, and much more to address before this part):

1. Tag-based attributes for resources (as in "meta-data"). Tag combinations can be leveraged to apply specific "rules". But the interpretation of those tags won't be a jump-table or vector-table. If I can't see the domain I'm debugging, neither will the devs who come after me!

2. Taxonomies suck for this kind of thing. Ontologies (what I see as Taxonomies with links) may be required in extreme cases, but just make sure that the full path is always "visible", not just a pointer to node a in the ontology graph. The project I did work on for some time was usurped by another that grabbed my attention (I was re-assigned). Too bad, I didn't get to follow this through. I don't think this is needed for my current project, but time will tell. (I hope not, as it's "elegant" but not at all "simple")

3. State tracking automata may be possible as well. Each request to execute rules stores the path through the rule set directly within the request. Easy debugging and visibility. I've not implemented this type of "rule" system before, but I'd like to give it a shot some day. Of course, writing rule paths is time consuming (even if in-memory) and may be overkill.

4. One (simple) example of a rules engine is a command-line tool's argument parser. Does anyone store this data in a jump-table? I don't think so, (maybe GIT or AWS-CLI, but I don't know as I've not looked yet). Most solutions have a parse pre-step and then a current-state result, usually followed by a "switchboard" type rule applier. But, that state is usually always available; I think that simplicity.

In short, for a given set of "attributes" which drive a "policy", I'd prefer to see the code rolled-out and explicit rather than deferring to some dance between a set of "rule tables" and a set of classes that get invoked in "special" ways.

Of course, I know nothing of your domain; so please take this all with a grain of salt. I've not worked with more than several dozens of "rule sets" before; you may be facing 1000's (and it sounds like speed is critical for you as well). Your rules may be transient, time sensitive and possibly even self-modifying (ugh!). If so, I'd make certain that the path through the rule-sets (if truly required) are well documented and verbose in their logging.

[adrianratnapala]

> * AOP for logging; ...

Can you expand on how this works in practice. Or link to an explanation? (I assume AOP means "Aspect Oriented Programming").

[Jupe]

Yes, AOP == Aspect Oriented Programming

Logging systems can usually be enabled by package/namespace, and frequently to the class level. I've used AOP to log out the trace of the code (not just the call stack), but can turn it on or off as needed. Authentication acting strange? Enable those logging aspects and browse the log to see where things are behaving strangely, and it may not be in the stack trace of the exception that gets thrown somewhere else. Logging methods and some arguments is hugely helpful; it's like automatic debugging.

A quick googling found: https://www.yegor256.com/2014/06/01/aop-aspectj-java-method-... and https://nearsoft.com/blog/aspect-oriented-programming-aop-in...

[adrianratnapala]

Ahh OK. So you mean fine-grained selection and routing of log messages.

I'd never thought of that as AOP, but I suppose it is. And I suppose AOP frameworks provide the machinery to do it, without the logging subsystem having to reinvent those wheels.

[ozim]

I found IOC mostly usefull in my projects. Mostly problem is that when business is saying "we need to have this thing generic", dev team goes ape-shit like public WhateverGoesInAnythingCanComeOut List<T>(List<T> love), where "generic" for buisness people is "copy-shit-paste" and change 2 things.