Never dismiss the dangers of over-engineering. While not as terrifying as under-engineering, how useful is a feature that exists only for the purity of design?
Over-engineering is a problem because it usually results in a much more complicated design than you really need and wastes a lot of time.
However, over-engineering and spending quality time thinking about the design and domain are two separate things and typically the more you think about the design and domain the less likely you are to over-engineer.
And Over-Engineering is something people only learn to spot once they've designed and built a few systems and had to maintain them for a few years. Its fairly easy to teach design-principle-du-jour but quite hard to teach about over engineering, KISS and YAGNI.
Developers tend to want to generalise everything because it makes it more interesting and has the appearance of good design. But generalising in one direction reduces flexibility in other directions. Better to copy and paste a bit at first and wait a bit before adding that extra extraction layer that generalises Foo.
I think a lot of it comes down to developers focusing on their own personal learning and development and not what is best for the current project.
Very few workplaces will let you work hard for 5 hours solving problems in the most time effective way and then spend the next 3 hours on your personal development.
On the other hand you can always spread the work over 8 hours and "experiment" with "on the job learning" leaving all sorts of sub-optimal solutions in the project. Then the next guy goes "Why on earth did Jim use X here?" and the truthful answer is because Jim wanted to learn more about X.
What we're looking for in design is simpler. Simpler becomes conceptually harder very quickly, because it means challenging a lot of assumptions and opening up some unknowns.
The tried-and-true implementation has a known initial cost and cost scaling factor as it's repeated throughout an application, while an experimental (and overdesigned) one is extremely sensitive to context and could become a net negative if it fails.
When we talk about "technical debt," I think we're describing something that isn't an experimental design, it's just "below" the known standard - it's faster to implement, but it cannot scale, and we already know that. Conversely, a standard practice is one that works and doesn't usually get pushed to its tolerance limits.
The corollary of this is that for any engineering project taking on a big enough problem, you have to assume some technical debt to ground the system in a place where each module can be tested as part of a whole: otherwise your work will go too deep into the experimental/overdesigned zone before it ever completely functions.
Yeah, in my experience, "no engineering" is the most abundant; everything is thrown together willy-nilly to code as fast as possible. I have come across very few real cases of over engineering.
Part of my point is that you don't feel over-engineering nearly as much. If it is under-engineered, it probably doesn't work, and at least has a list of pain points that are fairly obvious.
Over-engineered components will work (sometimes flawlessly), so you don't realize you've spent too much time and money on something until long after you built it.
Yes, never dismiss it. In fact a lot of the time over-engineering is just another form of technical debt, but it also has an upfront cost as well. At least the under engineered solution is cheap up front.
I would much rather come across something under engineered than over engineered.
However, over-engineering and spending quality time thinking about the design and domain are two separate things and typically the more you think about the design and domain the less likely you are to over-engineer.