[1337p337]

At the last startup where I worked, we built our own document store (in Scheme!), with exactly the same reasoning that causes other startups to over-engineer for scale before we had any users.

Premature optimization is a kind of over-engineering. Thankfully, it became so frequently hammered into new coders' heads that premature optimization is a Bad Thing that it's no longer much of a worry. It seems to have been replaced with premature scaling in the past few years, though. Anyone that could solve the problem of rampant over-engineering could potentially wield the awesome power whispered about in legends.

[hvs]

The problem is that over-engineering often has its roots in the very same drives that brought many of us into technology in the first place: the opportunity to work on something "really cool." As a developer turned manager, I've seen both sides of the coin, and it is a delicate balancing act. You don't want to completely stifle innovation (and morale) by being hard-headed about over-engineering. Conversely, you want to direct that energy appropriately so that you don't create a culture of "under-engineering" and unhappy developers.

[1337p337]

I totally agree. I may be totally wrong on this point, but it seems to me that over-engineering (in the forms of feature-creep, premature optimization, premature scaling, over-specification, planning for too many unlikely use cases, and all its other myriad forms) is often a tunnel-vision problem, a feature of the blessing/curse of engineers' tendency to focus really hard on a single thing.

If you look at the single-page source code to Plan 9's implementation of the cat command, and then compare it to GNU's, you can almost see it directly. The Plan 9 coders were looking more broadly at the OS and knew they'd need a cat command, while the GNU coders were thinking just about the Unix userspace, so when Plan 9 had cat, they continued hacking by moving to the next thing, but when GNU had it, they continued hacking by stuffing more things into cat.

Hackers love hacking, and this is great for the hackers and the users, but one rarely hears "This program is sufficient. I will consider it complete and write a different one until it becomes clear that the program is insufficient." (I don't exempt myself here.) Or maybe it's just too apparent when the designers forget it, so it stands out.

It's even more apparent on the web, where the default mode of a site is to see itself as a site rather than a component of the web at large.