I had a school assignment about writing a letter as if I were part of the Trojan War. I figured I'd learn Dactylic Hexameter and write it all in Dactylic Hexameter. Bad idea.
This is a project management problem, not a problem with too smart developers.
The problematic activities described here, overgeneralization and solving imaginary problems (aka Coding for A Rainy Day), are simply typical newbie mistakes.
Otherwise highly competent developers almost always fall into this trap early in their careers, so the important thing is to have someone experienced in charge of the project who can veto their weirder ideas until they mature.
I think every software company should hand out a copy of "The Pragmatic Programmer" to all new hires. It gives much good advice on how to not overcomplicate things.
- Designing too much flexibility at the start of a project and regretting it when it takes ten weeks to build a prototype.
- Designing too little flexibility at the start of the project and ending up living with unmaintainable cruft.
- Realizing that all projects are hell and planning on a rethink/rewrite/refactor when the real requirements become clear because no one can predict the future, so why not hold some contingency for the inevitable.
Actually, EC had a couple of rather sharp project managers who unfortunately were never really given enough authority to overrule the bulk of the engineering team. EC as a company was the poster-child for "too many mad scientists, not enough hunchbacks." The items Chip describes as being worthwhile Ph.D. thesis projects had their own color on the PERT chart (purple I think...) -- if that is not a sign of impending doom I don't know what is. OTOH, it was great to work with a huge collection of smart folks valiantly trying to impose their vision on the world.
I think there's another implicit point as well, echoed by a lot of more recent startups, which is to fail early and often. If you're going to make the engineering decision to go down a path which will take a year+ to validate (i.e. developing a "PhD thesis" of a filesystem), then you'd better have a very compelling reason to do so over the current solution.
Conserve your best thinking for emergency use only? (Something about that seems wrong, even though the systematic logic of leaving a tolerance for error is sound.)
No, no. Conserve cleverness for emergencies. Logical consistency is the best policy: most of the time, you should trudge along, applying existing solutions (2 wheel drive). The only time you should be clever and try to do something innovative is when your traditional models fail you. (4 wheel drive).
On the other hand, this style of thinking has some risk of demotivation. It collapses varied results into a static judgment prone to a 'fixed mindset': no matter the inputs of intellect/talent/effort, the end result is you hit a limit. So smile at the insight but remember which limit you hit still matters. The deeper snow is more fun.
Anybody can rationalize anything to his own satisfaction, regardless of IQ. The problem with being smart is that there are fewer people who can see through your rationalizations. The skills of dissasembling your own cognitive biases and testing your ideas against reality grow in value as your IQ increases.
"Anybody can rationalize anything to his own satisfaction"
... but smart people tend to require a bigger rationalization to be satisfied. And they also tend to be more capable of creating a rationalization of that magnitude.
The question is whether they've developed the habit of "reality testing" and discarding flawed rationalizations.
A rationalization means to come up with reasons to support a belief, idea or action that one knows (or could know) is not actually rational. It is a form of dishonesty mimicking rationality. The issue with smart people is not that they rationalize (i.e. are dishonest) but they are able to come up with sincere reasons, often very convincing reasons, for their beliefs. This makes it very difficult to get them to question their premises.