Long ago, in the mists of time, (1987) I wrote a program that worked with hand held computers, barcode scanners, and was meant to be run by folks working in a power plant.
I wrote the code, and got it working in about 2 months. It did everything in the spec, and the customer (Russ) loved what we had, but it turned out (of course) there were many things missed in the spec.
A week or two later, we worked out a deal for all of their power plants, provided I would work with Russ to make sure it did everything they wanted... it would take about a year, in the end. Russ taught me everything I needed to know about users and how they think, in a direct and very effective manner.
He was the assistant plant manager, so he would pull a random person into the office, and say to them "I know you're not a computer person, I want to you to do X,Y,Z... but don't worry... if anything goes wrong, it's not your fault... it's Mike's fault (Russ points at Me). He then tells me to just watch... It took exactly 1 minute to learn the first lesson... there should always be "Press F1 for Help" somewhere on the screen.
It was a very instructive and productive year. I've carried those lessons from his forcing function with me for decades. I love telling the story, thanks for listening.
> Try to add support for a completely different database. Details of your current database that have leaked into your data layer abstractions will soon become obvious
Well, details of Postgres have leaked into my database queries and schemas, but it was a conscious decision to use Postgres and its features. Sure, it seems nice to be able to swap one database for another, but you lose out on a lot of what a database can do if you stick strictly for the lowest common denominator of features. I use Postgres partly because of its feature set, so I am going to use these features. This does mean that its unlikely I will ever run my software against a different SQL database, but I'm ok with this.
I guess an important note is that you should be aware of it and it should be a conscious decision, rather than something that crept in over time.
But I suppose that's a tangent and not the articles point. Forcing Functions is about unearthing the brittleness that has crept into a codebase over time and I absolutely agree that is a helpful and worthwhile exercise. I've seen plenty of codebases that were meant to be database agnostic, but porting them was still not painless.
Yeah I don’t get that obsession with “pluggable databases”. Abstractions have a real cost, and they typically complicate a lot: I would advice to just make sure you centralize the access to your database in a single module / class / whatever, but other than that, swapping out databases seems like a non-goal to me.
How often do people really migrate to a different database? Even when doing so, migrating the existing data always is a lot more tedious than migrating the code, in my experience.
Yeah I don’t get that obsession with “pluggable databases”. Abstractions have a real cost, and they typically complicate a lot: I would advice to just make sure you centralize the access to your database in a single module / class / whatever, but other than that, swapping out databases seems like a non-goal to me.
I tend to agree, as long as the database you’re using is of the free and open source type so it’s a low-risk dependency.
The “pluggability principle” holds more strongly for higher-risk dependencies, IMHO. For example, if you’re working in an online environment and you can’t readily integrate a new payment processor or messaging service or whatever, your existing dependency starts to look like a single point of failure that is not under your control. Some careful abstraction of the essential features and avoiding dependencies on the peculiarities of any specific platform can be a valuable safety blanket if your external dependency catches fire one day.
It's an example of what economists call "The Hold-up Problem", as part of "Incomplete Contracts".
Basically, it's impossible to write a contract to cover all situations, so eventually any two parties will need to renegotiate. If one has leverage over the other, they can do stuff like raise prices or simply refuse service. One factor affecting that leverage is "asset specificity": given an asset that's covered by the contract, is it reusable in different contexts or is it too closely adapted to one purpose?
The literal textbook example is a steel foundry and a railroad. The steel company needs a spur line to its foundry to receive raw materials and send out finished steel. The railroad company can run a spur from its main line to the foundry.
In this case the asset (the spur line) is extremely specific. Without the spur line, the foundry is worthless. Without the foundry, the spur line is worthless. The balance of leverage then comes down to relative costs and gains. The railroad will hold the upper hand, because closing a spur line isn't existential, but it is for the steel foundry.
In the software context it shows up as a "contract" between your software and a database. You can use an ORM to try to insulate yourself, reducing asset specificity, but that decision has costs (eg, specific DB features you must forego). You can choose to increase asset specificity by adapting to only one DB and fully utilising its strengths, but you may face a future hold-up.
Note from the train example that hold-ups needn't be "pay me more". They can also be "I'm walking away". So a hold-up from Oracle will be "pay me more", whereas a hold-up from an obscure database with one developer could be that they simply quit, die or otherwise become incapable of working.
I agree, I see those patterns in the wild a lot. You have a Microsoft shop using .NET and SQL Server yet the devs still abstract the data access layer. I think those patterns just became common 'just in case' they were needed. I find refactoring tools like "extract interface" take care of that so there is not need to write the abstraction until it's needed.
On the other side, I worked on a web app that supported multiple db vendors, we did the classic DAO pattern which worked well. You still get to use custom SQL for each database if you need to.
We tried an ORM at one point which worked out well. It was the same web app and we moved moved some DAO code to the Java Persistence API. We could then build the data access code and include it into our desktop (Mac Windows) and plug it in to a local DB (Derby).
In that case, once JPA was working, the pluggable database was allowing us to save on development costs.
> How often do people really migrate to a different database?
Well, I have been part of a migration from MySQL to MariaDB, which was a lot more effort than one would expect given that they're meant to be more or less the same thing. It was a ton of effort and the abstracted ORM logic didn't actually help with this.
So if it doesn't really help that much for a simple case like that, then it doesn't seem like there's much point in my opinion, as porting is going to be work either way (in a non-trivial application with non-trivial data access patterns, at least).
I don't remember, it was ~4 years ago and for a different company. I believe there were a small number of incompatibilities somewhere (composite indexes maybe? I remember doing stuff with them but don't remember why, could have just been for query optimisation) and a few areas where the performance was degraded (again, I think around indexes, but I could be mistaken).
It wasn't a big deal over all, but it wasn't simple plug'n'play either and required some porting work.
Should library code really be relying on a database? If its a database library, then ok, maybe it can stick to a subset of database features that works across all supported databases, but in that case, testing against these databases is no longer a "Forcing Function", but a core part of the libraries test suite to ensure correctness.
The same goes for platform code, really. If you are advertising your platform to work with databases X, Y and Z, then you really need to be testing against those and not just as a Forcing Function to see if any brittleness crept in, but as a core part of ensuring your platform does as advertised.
> Should library code really be relying on a database
Yes, many libraries need, or are enhanced by, persistence, whether that's a double-entry accounting module, a headless cms, or an e-commerce engine.
> you really need to be testing against those
Library authors need tools and guidance, not arbitrary constraints and high-watermark QA demands. Most of us have one or two DBs that we work with day-to-day but still want libraries we contribute to be broadly portable across any of the backends the framework they plug into supports. e.g. I can't maintain a test suite vs Oracle when I don't license it, but I still want to know that It'll Just Work if someone uses it in such combination, or that it's at least close enough they'll not have trouble making it work.
More broadly I think the myth of testing every supported combination leads to a brittle mindset of saying "not supported, won't help", especially when systematised in a commercial environment, and to me it's the antithesis of good engineering.
Well, if your library can use the "lowest denominator" of supported databases, then there's nothing wrong with doing that. I'd still argue that you don't really support something you haven't tested against, but I guess this is pretty off topic. My overall point was more that sometimes being locked into a single database is a conscious decision because you want to make use of everything a database has to offer.
In the context of the article, for "Forcing Functions", I absolutely agree that switching databases is a useful way to find weakness in your solution.
I don't significantly disagree with you, but let me take the devil's advocate approach.
Adding support for a different database doesn't mean restricting yourself to the lowest common denominator. It means using different techniques, more appropriate for the different database, that may optimize other parts of your data access and modification path, while pessimizing stuff your current database does.
More importantly, it means extracting hard database dependencies like raw SQL or custom ORM fiddling from your business logic and entities, and pushing them behind a module or service boundary.
Raw leverage of the database, if it's dispersed throughout your application, will limit your ability to change your schema (e.g. denormalize an attribute, split or join tables, convert a parent-child relationship to embedded JSON or vice versa) and address performance problems as you scale up. It'll also stop you having a single point of data access where you can partition or duplicate your data into different stores with different capabilities more suited to their access and modification patterns. These kinds of things become really important when the database becomes a bottleneck in your system.
Just checking if I understood your point: for the purpose of Forcing Functions, running against a different database than the one designed for (and therefore where the tradeoffs may be different and things may run inefficiently) is still useful because it helps unearth design flaws, corner cases or brittleness?
If so, then, sure , I agree with you. Not all reliance on a target database is actual features that don't have an easy or direct way to port.
More that being forced to separate church and state, forced to keep business logic separate from database manipulation, and forced to go through an abstraction interface, then lets you leverage the interface later if and when the database becomes a bottleneck.
It's a devil's advocate position. I actually think it's unrealistic and probably not worth it at the time. The forcing function is what's useful, not the database portability per se. And whether it's actually useful depends on being so successful that you have so much data that your regular RDBMS can't cope with some access or update patterns. That's a hefty bet early in anything, and probably doesn't make business sense, until it does, then you wish you did things differently.
(Not coincidentally, it's a position we're in at my workplace; the database has hit its limits and we're needing to use hybrid approaches to hit latency NFRs.)
Agreed. It makes little sense to try to write your SQL queries to run fine on multiple different DBMSs. Different SQL DBMSs should be treated as different languages.
Presumably the idea is based on an analogy to code portability: it can be good to ensure your C++ code compiles fine with multiple different compilers. Really though, it's more akin to writing code that compiles as both C# and Java; clearly madness.
I took this to mean be able to swap in and out your persistence layer.
So my app can make calls which will persist data to a SQL database, or I can swap that layer out with another that persists to NoSql storage in the cloud. Possible because the persistence layer exposes an API or interface that is agnostic to the actual implementation of the layer
At an agency, we used to run our web apps on some crappy 08 model laptops running on a gig of memory with outdated browsers. If the webapp ran there without major hitches, it was considerd good enough. It made everyone on the team think hard about optimizing even before a single line of code was written. It really did force excessive simplicity and not jumping on new libs/frameworks just because we can.
If a tree falls in a forest with no-one around, does it make a sound?
While the list of techniques here look like excellent ways to uncover unknown unknowns, be sure that it's actually valuable for you to resolve these issues.
In almost all cases the goal isn't to create perfect software, it's to create effective software - and sometimes the ROI on these unusual cases doesn't stack up (or doesn't stack up _yet_).
> Delete the project from your development machine, clone the source code and set it up from scratch.
Very good advice. This was obviously never done in my org, leading to newcomers (me included) wasting weeks to get started on some projects. Once it was fixed, a newcomer can start working in an hour.
actually having a newcomer install everything and fix the documentation while doing it is even a better option, cloning has some of the dependencies but other might have been installed by the developer for example global packages for NPM or Python, setting of system wide environment variables etc.
This happened at my org... 8 separate times. It was always slightly wrong each time, even after it was fixed. I think it's better now, but I never considered doing it myself as an established team member.
Though I once inherited a large enterprise code base that I had to study and build a dev environment for pretty much all by myself. I had maybe 2 calls with the original team and a couple of emails but was mostly by myself just figuring it out. It was a pretty incredible experience and taught me a tonne about how the system worked and was put together. This helped immensely when we spun up a team to work on it. So, I get where this article is coming from due to that experience, but didn't think to do this kind of stuff on purpose.
Unless you're hiring like crazy, it's never going to be and stay perfect, but hopefully with fixing whatever issues each time you stay relatively on top of it, and it's never too much work.
Who has time for this? I've always been on under resourced projects where every hour had to be signed off. Everything is a rush and quality is required but never budgeted for.
It's a bit of an argument for "less haste, more speed".
As an example, I've always found with CI (continuous integration) servers that setting them up on day 1 of a project takes almost no time, but trying to set them up 3 months (or later) into the project seems to require a lot of time. Once a CI server is set up, it invariably improves both quality and productivity significantly, they start yielding dividends on their time investment very quickly.
If management claims they can't afford the relatively small amount of time required to set up a CI server, then I would argue that the lack of time only strengthens the need to have it done sooner to enable the project to move faster.
For onboarding documentation, finding random time may be hard, but it's almost free if it's done by a new member as they join a team. As they get their environment up and running, they just need to document the steps as they went along. Make sure it's committed to the same source control repository, readme.md seems to work well for this. It's fine if it's initially very simple, just an unformatted list of steps in plain text is a great start.
If someone is adding new technology stacks which would would affect the onboarding document, they should quickly add it at that moment, while possibly improving it a little by adding a little formatting. Future new team members should also be encouraged to improve the documentation based on their onboarding experience. Over time the document becomes quite refined and easy to keep upto date.
That doesn't address all their points, but it's a start and I hope it's helpfull.
HN resorts to the "just quit your job lol" far too easily. people are in all sorts of different situations. you don't know if they are struggling to find work and this is the one shot they have been given. you don't know if they are in extreme demand and get paid well to fix bad situations like this. you just don't know. and telling people to blithely change jobs when they can't or don't want to isn't helpful
Indeed, I encounter this anti-pragmatic personality a lot in tech people.
It's an extreme intolerance to imperfect circumstances: a preference for nothing at all over compromise.
The issue is that tech attracts the mathemetically-minded who reasons from universal principles. Rather than the empirically-minded who start with cases, and abduce to provisional principles from those.
To a aximoatic mind: when a universal principle is violated, the situation is declared Bad.
To the case-base mind: when a tolerable situation seems to violate a principle, declare the principle Inapplicable.
Of course both types of thought are helpful in different contexts, I suspect 'the management of one's life, day to day' however, should be a matter of case-base reasoning to rough principle.
Okay, that's a reasonable universal principle, but in this case it's Inapplicable. To quote the OP:
> Everything is a rush and quality is required but never budgeted for.
It doesn't really sound like this is a situation where the principle is wrong, it sounds like this is proving the principle correct: you get what you pay for.
I think the "you get what you pay for" is the kind of heuristic reasoning I'm advocating. It's essentially balance/tradeoffs/etc.
That trading off thought process isn't the same as the ACCEPT|REJECT process of the axiomatic mind.
I think a person who says "quit" is really saying that the very question of trading off heuristics of value isn't applicable.
It's a bit like Poisonous|Edible, or Gold|NotGold. There's nothing to be traded. A Copper apple is Poisonous and NotGold. No two ways about it.
That reasoning only works when the concepts (Gold, Edible, etc.) are natural kinds -- or otherwise disjoint and universal classifiers.
In life, situations fall both into the ACCEPT and REJECT categories, into both GOOD and BAD, into both VALUABLE and WORTHLESS. These concepts are heuristic ones, and not disjoint & universal.
The attempt to apply this "disjoint, axiomatic, ..." reasoning to life is a recipe for catastrophe.
EDIT: my point about principles vs. cases, is that i take: def. heuristic "a resemblance amongst cases"; and def.., principle "a universal rule which disjointly classifies cases"
..ie., a slightly more extreme meaning to "principle" than is in general use
I think it is perfectly fine to remind people that looking for something better might be an option. Even if you have good reasons to stay in that situation, this reminder will do you no harm.
The reason might be they've been looking for a different job for a while without luck, and the harm might be... pretty depressing having someone tell you how easy it is?
I sort of agree and am just playing devil's advocate, but I also think everything of 'there's better things out there you know' has probably already come up thread just before the person complained of their situation not being as good - they know better's out there at that point.
The reference to bugs being more cheaper to fix the earlier they're found is one of those claims that gets passed around without much attempt to look at the supposed sources. I believe the most common citation trail bottoms out in a study that no one has access to anymore, and the whole thing seems dodgy. One link: https://www.techwell.com/techwell-insights/2013/10/what-does....
Checkout the book "Accelerate", it covers a lot of operation excellence practices supported by data, including shortening the iteration cycle (e.g. discovering bugs early):
I don't buy the "bugs caught early are cheaper to fix" paradigm. Most bugs are caught early without special precautions, but special precautions can be quite expensive. Most software doesn't have truly catastrophic failure cases.
If there's a bug slipping through, someone will run into it, report it, and it'll get fixed. If nobody runs into it, the bug doesn't cost anything.
This is the most economic way to go about it, which is the reason why pretty much all successful software is kind of buggy. We all like to complain about it, but then we don't want to wait an extra year for the next version either.
At the other end of the spectrum, if you need really reliable software, the solution is not to eliminate all the bugs, that's impossible. Even with perfect software, hardware can fail, bits can flip the wrong way. The solution is to make sure that errors can't bring down the airplane.
I don't think so. When one is fixing bugs locally before the thing goes into production one can use each and every debugging strategy under the sun to find the bug. If it runs 10 times slower because of generating massive logs that is acceptable when debugging locally. In production one is constrained to debugging techniques that do not disturb production. This constraint quite often makes debugging times longer by factors of 10 if not 100 while at the same time a customer is getting somewhat impatient.
Also, the bug that would be discoverable locally but not in production is a bit of a strange animal. It does happen for cases where the developer has a better idea than the user what should be happening so the user will not even notice the bug. That sounds more like features that have been specified in too much detail than true bugs, though. More commonly everything that can go wrong locally will go wrong in production sooner or later. And in production you will, on top of that, hit all of the bugs that occur once every month and that only occur if the order of inputs is a bit strange while the database is under some load. If you have not done all possible debug/test work locally the application will hit production when there still is a lot of debug work to do and there will probably be so many problems that they even start interacting with one another and produce some really 'interesting' failures.
This is just pushing testing to the customer. Why would the customer report a bug instead of switching to a less buggy product?
Why do you think the customers are going to report bugs? I can't say that I've ever attempted to report a bug on an iPhone app, Windows, etc. It's like yelling at the wind.
> Why would the customer report a bug instead of switching to a less buggy product?
All products are buggy to varying degrees. As a user, you can't easily tell if another product is "less buggy". Would you risk investing time into figuring out another program that may turn out just as buggy? No. You most likely move on with your life.
Once users are invested into your product, it takes a lot to make them switch. Every single piece of software that I use regularly is either extremely simple or somewhat buggy. I haven't switched once because of it. It's a nuisance, but not a dealbreaker.
That's not to say that I like this situation, of course I would prefer software that doesn't have bugs and glitches, but I also prefer software that exists today, not tomorrow.
The market has spoken: worse is better.
> Why do you think the customers are going to report bugs?
If it's an important bug, somebody will most likely report it. If it's not an important bug, not having it reported is most likely not important either.
> I can't say that I've ever attempted to report a bug on an iPhone app, Windows, etc. It's like yelling at the wind.
So, did you switch to Android or Linux/MacOS then? Is the grass really greener on the other side?
The cynicism in this comment really bums me out. You think it's ok to give customers a buggy product, and expect them to report the bugs, because they don't really have a choice and every other product is buggy too. You say you don't like the situation, but you're doing nothing to challenge it either.
Is your motto "When life gives you lemons, make a market for lemons?"
Fair enough, it's arguably cynical to tell the children that Santa Claus isn't real. However, we're all adults here.
The highest-quality software can not win in the software market. This is evident from the software that is out there owning the market.
Quality is a trade-off, if you spend too many resources on it, you can not compete. Catching those last few bugs takes exponentially more effort.
Moreover, there are snake oil salesmen at every corner, telling you that if only you adopted some methodology, your defect rate would plummet. It's easy to get lost in that, not actually delivering a product.
I wrote the code, and got it working in about 2 months. It did everything in the spec, and the customer (Russ) loved what we had, but it turned out (of course) there were many things missed in the spec.
A week or two later, we worked out a deal for all of their power plants, provided I would work with Russ to make sure it did everything they wanted... it would take about a year, in the end. Russ taught me everything I needed to know about users and how they think, in a direct and very effective manner.
He was the assistant plant manager, so he would pull a random person into the office, and say to them "I know you're not a computer person, I want to you to do X,Y,Z... but don't worry... if anything goes wrong, it's not your fault... it's Mike's fault (Russ points at Me). He then tells me to just watch... It took exactly 1 minute to learn the first lesson... there should always be "Press F1 for Help" somewhere on the screen.
It was a very instructive and productive year. I've carried those lessons from his forcing function with me for decades. I love telling the story, thanks for listening.