[taeric]

How did that work out for the Kernel debates? :)

I don't think anyone disagrees that a "well designed and written" program of loosely coupled pieces has advantages over a monolithic one.[1] The debate is really over which is easier to do. And, the argument you are responding to is essentially, that it is easier to abstract out parts after you have done it in whole a few times. I know, personally, that that is a very compelling argument.

[1] Well, there probably is some debate on the feasibility of making things as loosely coupled as you would like. Back to the kernel debate, how many microkernels have survived with the device support of linux?

[spand]

The kernel debate is a different one.

Software can easily consist of loosely coupled pieces as source code and be compiled and run as a single monolith with hardly any performance loss at runtime (versus coupled source code).

[taeric]

Maybe I misrepresent the debate, then. My understanding is that the debate was that there was no future in a monolithically sourced and run kernel. Linus took the position that while that had a certain appeal, he just wanted an operating system he could use. If anybody had managed to deliver on the microkernel dream, he would probably not have started the linux kernel.

That is to say, that there is appeal to the "loosely coupled" dream of a software solution. Not just in source but in execution. However, there is the reality that this is very hard. The contention in this thread is that to think you can start in the loosely coupled set of parts is very ambitious.[1] It isn't that it is a bad goal. Just that it is akin to wanting to score well in a marathon without first running a few smaller races.

[1] Unless I am misrepresenting that, of course.

[jerf]

The real problem with the debate here is that a kernel is a place where performance is in your top two concerns, fighting it out with correctness, and among other things beating out "effort to create" and "skill level needed to create". If your software does not have performance as its absolute #1 criterion, and you care about the effort it takes to create it and the skill level needed to create it, you'll probably want to go back to easily isolated pieces that can be tested and understood without the whole system being understood, and that may not perform the absolute best that they could. (Although I find this software doesn't produce slow software on its own; at most it costs you a few more pointer traversals than you may like. Slow software is IMHO far more likely to come from highly coupled programs that everyone is terrified to optimize lest the whole thing come apart.)

Trying to use the kernel as a template to guide all software development is not a great idea.

[taeric]

I still think the loose concept of "loosely coupled pieces" is a massive hand waving over the difficulty in making something multiple pieces. It is typically the goldilocks search. When do you have too many pieces, and when do you not have enough?

I have yet to see a prescriptive approach to this that works. About the best I've seen is the holistic iterative approach. First make something, then look to see where you can isolate changes and make them. Repeat. If this fits a model of TDD, it is new to me.

[jerf]

I disagree that it's massively difficult. I think it's a skill that has to be learned. As I get better at it, I become faster putting together something loosely coupled than tightly coupled... because while writing something big and monolithic may have a momentary short-term advantage, when it comes time to, you know, make sure it works, correctly, my system is a lot easier to verify, test, deploy, and ship than the monolithic one.

Programming speed is not the only consideration when it comes to shipping software. Squishing something together as rapidly as possible may shorten the programming time (and then only for smaller systems), but only at the cost of shoving the time into all the other phases, usually at a ratio greatly in excess of 1:1!

In other news, programmers are generally pretty bad as estimation, and this is probably related. I suspect the estimations for the "squeeze something together" part are pretty good overall, it's the rest that breaks down.

And again, to be clear, I'm not disagreeing that it's challenging. I'm saying that rather than being fundamentally challenging in a way that can never be made easier, it is a skill that can be learned. That makes for a very different cost/benefit set than a task that is fundamentally difficult. And, frankly, few developers are taking the time to learn it; far more are sneeringly dismissive at the skills that are required to learn this. Rather a shocking amount of our "structure" in programming is still just covering over cowboy programming with terms management can get behind. I think XP actually avoided this, but the average bastardization of XP is a thin patina of words over cowboy programming.

[taeric]

I don't disagree that it is a skill that can be learned. Quite the contrary. I just feel that likely the best way to learn this skill is to first build a few systems that aren't loosely coupled first. Consider the analogy for building cars. Before you try and build a continuous transmission system, first get a direct drive one working. Then, determine what would need to be messed with to put basic gearing in place....

Now the major trick here is that this breaks down in categories that are effectively already solved. Which is why many of the examples are obnoxious to the point of unhelpful. If you know how to break something down to where it is loosely coupled parts already, I feel you should definitely do so.

[memracom]

The Linux kernel is not monolithic in the sense that people are talking about here. There are modules or parts in the Linux kernel that are composed using carefully crafted APIs. The kernel difference is that the design goals are different from most applications and that Linux leverages every possibly way to integrate software components on a von Neuman architecture. It goes well beyond what you normally do in a business app.

Linux is well designed and you can learn from it, but in order to get value from that study you need to be a skilled C programmer and at the top of your game. Therefore it is a bad example for people who mainly use other languages.

In additon lets not forget that the SOLID principles, DRY, YAGNI and so on, are not hard and fast rules. Every extreme programmer will regularly violate those principles. The purpose of the principles is to guide your work, to make you see clearly what you are doing, so that when you violate a principle you do it for a good reason.

[taeric]

Mayhap I am misrepresenting the origins of Linux. Simply put, it was less rigidly modular in construction than it could have been at the beginning.[1] Indeed, I think it is a perfect case for the argument of "first get it done, then figure out how to make it modular." Probably a better argument for keeping the model such that you can keep the full picture in your head when working on it. Not sure.

[1] Consider also this lovely thread: http://www.realworldtech.com/forum/?threadid=65915&curpostid...

    You can do simple
    things easily - and in particular, you can do things where
    the information only passes in one direction quite easily,
    but anythign else is much much harder, because there is
    no "shared state" (by design). And in the absense of shared
    state, you have a hell of a lot of problems trying to make
    any decision that spans more than one entity in the
    system.

[memracom]

When people say that the Linux kernel is monolithic they really mean that it is not a microkernel architecture. In microkernels the designer says "Having a minimal microkernel with lots of separate modules is good therefore we will do everything that way". Linux simply says that modularity is good therefore we will use the appropriate integration technique at the appropriate times.

In the Linux kernel, you use printk to print messages on the console. That is modularity. There are device drivers. That is modularity. There is a range of loadable/configurable kernel modules for many things and these used to be more visible when more people would configure and build customized kernels. The Linux kernel has far more of these modules than earlier OSes that I used (TI DX10, UNIX 6th ed., Xenix, 3B2 Unix, SCO UNIX). The Linux kernel is linked into a monolithic binary that runs in kernel mode, but it is composed of many modules, some of which are integrated at link time and some of which are loaded dynamically (lsmod).

There is a good reason why the kernel is more monolithic than a business app, and that is that the kernel is doing a vastly different job at a vastly different layer of abstraction than a business app. You might also note that there are still lots of jobs for C programmers but most of them mention "embedded systems". That's what the Linux kernel is, a big featureful embedded system.

Perhaps some day someone will write a book on integration and cover all the different ways in which functionality can be integrated to produce an application. Most developers lean far too much on only one way of doing it, i.e. the link editor. For most apps, loosely coupled integration techniques are more valuable.

[taeric]

I don't think these terms are as different as you seem to be implying. When someone says "loosely coupled modules" they don't mean "printk" or similar functions. They mean such joys as power management and thread scheduling. These are somewhat modular in the kernel, to be sure. Are they so modular that you could TDD one or the other? My last understanding was not really.

Consider, you can have a device driver that runs fine "on its own" but crashes when run with another driver loaded. This is almost canonically the opposite of loosely coupled modules.