[nostrademons]

Point 2 is something I've heard Jeff Dean strongly suggest. Write microbenchmarks for all the basic operations of the "black boxes" you'll be using to build the system. That way, you can do quick back-of-the-envelope calculations to estimate the performance of the system before actually building it. You can explore the problem space much more quickly when you can simulate the important characteristics in your head instead of having to actually type them out into an editor.

Point 5 is something my former manager once told me. He'd done RoboGames or something like that as a grad student, and he found that invariably, the teams that won were the ones who had built the best debugging infrastructure into their robot, so that when it didn't behave as expected, they knew exactly what was wrong.

This is as much a property of the program as of the programmer. The best programs usually have debug information built into the structure of the program itself, along with an easy UI for viewing that debug info so you don't have to stare at log lines all day. Once a couple programmers start doing this and everyone else realizes how much time it saves, it becomes part of the culture of the organization, and everybody starts designing programs with an eye toward "Will I be able to figure out what's wrong if this doesn't work?"

[delian66]

Please, would you give an example of an open source program with such infrastructure built in ? I'm eager to study such programs to pick up insights.

[nostrademons]

The closest open-source example I can think of is --verbose mode on many UNIX utilities, though that wasn't really what I had in mind. Developer Tools in Chrome is another example of the principle - think of how much easier it is to debug Chrome than IE6 - but again, not quite all the way. Imagine using Developer Tools to debug all of Chrome, eg. Webkit rendering (overlaid on the rendered page itself), resource parsing & scheduling, JS JIT compilation, etc, and not just the pages that execute within it. (Maybe this is possible, I'm not familiar enough with the Chrome project to know all the debug flags available.)

[swah]

That's a great insight/tip - I wish there was a book with hundreds of those...