[michaelochurch]

Yes yes yes yes yes yes yes yes yes. This is absolutely true.

The program-to-programmer relationship deserves to be many-to-one. It's a rewarding way to do things. You solve a problem. You add value. It's Done. You may have to go back to a program later to add features, but you don't end up with massive codeballs.

When the program-to-programmer relationship is inverted and becomes one-to-many, you get the enterprise hell with no feedback cycle, terrible code, and unnecessary complexity. It's not rewarding. Problems are never solved and software is never Done. Requirements are "collected", bundled into an incoherent mess, and delivered to bored, underachieving developers who never get to see their programs actually do anything.

Large problems that require more than one person need to be solved with systems and given the respect that systems deserve. Single-program approaches are a denial of the complexity (that comes whenever people have to work together) and a premature optimization.

I wrote about the political degeneracy that this creates: http://michaelochurch.wordpress.com/2012/04/13/java-shop-pol... . But it's unfair to associate it with one language. It's not that Java is any more evil than C# or C++. Any company that calls itself an X Shop is doomed.

There are cases where large single programs deliver value. For example, most people experience a relational database as a single entity. There are a lot of requirements (performance, persistence, transactional integrity, concurrency) that are technically very difficult to meet and all have to work together. I will also note that it has taken some very smart, very well-compensated, people decades to get that stuff right. The quality of programmers who tend to stick around on corporate big-program projects is not high enough to even attempt it, though.

So why is big-program development winning? There are a couple reasons for that. First, it gives managerial dinosaurs the illusion of control. If programs are Giant Things that can be measured in importance by "headcount", then executives can direct the programming efforts... which they can't do if the programmer's job is to go off and independently solve technical problems they deem to be important. Second, big-program design gives a home to mediocre programmers who wouldn't be able to build something from scratch if their lives depended on it but who, in teams of 50, might be as effective as 0.37 good developers. It's about control and a failed attempt to commoditize programmer talent, but it doesn't actually work.

[anon1385]

>So why is big-program development winning? There are a couple reasons for that. First, it gives managerial dinosaurs the illusion of control.

So why do larger programs 'win' in the open source world as well[1]? Pop psychology about management doesn't seem sufficient to explain the phenomenon (although I'm sure it is a good way to sell a '101 habits of highly effective managers' book or get paid to give talks about management). Large systems are large, breaking them up into smaller pieces doesn't change that, but it makes navigating the code base harder (although I assume you don't care about that since judging from your blog posts you don't think tooling is important). It makes your interfaces less malleable (can be good can be bad), and moves a lot of communication to places where the compiler can't warn you about mistakes (again, if you don't care about tooling I guess this doesn't matter… but I would argue that this is bad).

It seems to me like systems of many small separate processes is basically dynamic OOP. Everything is late bound, dynamically typed and async. It's easy to make changes and also easy to break things. You can argue that this is better for certain problems, but I don't think it's universally better, and the community seems pretty divided on the issue too: look at the popularity of Go, statically typed and building concurrency into the language rather than using the OS like in the older C world.

Aa an aside; surely the web developer community is eventually going to grow tired of talking about how terrible Java is and how $idea_of_the_moment is good because it's 'not java'? As an outsider the obsession seems extremely unhealthy, and leads you to bizarre places like arguing against automated refactoring or interactive debugging or static type systems just because those things are associated with Java. I guess to maintain credibility I also need to point out that I don't and have never used Java…

[1] Firefox/Chromium vs uzbl, gcc/llvm/clang vs pcc, gdb vs printf debugging, sqlite/mysql vs directories and plain text files, perl vs sed/awk/grep shell scripts, emacs/vim vs ed/notepad etc etc.

[chipsy]

I think async architectures are worthy but miss something important - the idea of "typed connections" between processes. Actors passing arbitrary messages doesn't make for an efficient "assembly line." Similarly, the Unix-style concurrency is tied to a single low-level protocol, which is not rich enough to adequately describe all data. It described "enough" of all data for 1970's era tasks, but our needs outgrew that.

When you set out to architect a customized, typed, async architecture, you end up with the "flow-based programming" style which has been captivating me recently. It tends to reduce to two events per component: "start" and "stop." The "integrated program" appears in a tiny top-level definition that sets up the components and connections. Components are relatively small and reminiscent of "pure" algorithmic code. Where synchronized behavior becomes essential, flows can be split into stages of processing and kicked off in a sequence.

This particular style has had a lengthy history of reappearing in numerous domains under various guises, and it has demonstrable effectiveness, but it can also feel alien and more "mathematical." The main issue is that has a lengthy design/prototyping time of weeks to months, and the initial complexity of the system looks high because you need a decent number of components to do anything substantial. This really, really goes against the "move fast and break things" mainstream, even within open source - everyone wants their project to just _instantly_ accelerate from 0 to 100 in terms of progress, and we've put most of our efforts into a toolchain that makes it easier and easier to do that.

The best remedy at present seems to be to embrace asynchronity, embrace static types, and maintain faith in both - i.e. to have a lot of discipline.

[michaelochurch]

I regret that I have only one upvote to give this post.

[michaelochurch]

On the anti-Java bias, I think the issue is that there's more than one Java culture. There's the horrid commodity developer culture, but that's not the language's fault.

I'm actually a pretty big fan of static typing. You don't get static typing's main benefits in C++ or Java, though. You have to use a language like Haskell or Ocaml, or the right subset of Scala, to see the major benefits of that.

Open-source is a bit different because people choose whether they contribute to a project. The quality of code in the active open-source world is leagues above what you find in typical enterprise codeballs, because of survivor bias. No one has the authority to mandate that code be maintained by others, so the messes are cleaned up by people who actually care, not people slogging through it to keep a paycheck coming.

The big-program methodology of the corporate world is the evil. In FOSS, the major projects are an unusual set-- code-quality at a high level just not seen in the for-paycheck commodity-engineer world and large because of success-- rather than the reverse. There's a survivor bias that occurs because the best projects are the only ones people pay attention to.

The corporate world is screwy because projects become large or small based on political reasons that have nothing to do with code quality. In the FOSS world, code-quality problems related to growth will be self-limiting because no one has the authority to "force" the program to grow.

[anon1385]

I should mention that I've never worked in the corporate world, so my reaction is in that context. I can't talk about the corporate side of things since I've never experienced it.

Also yes static typing in Java does look pretty cumbersome. Personally I'm hoping that Rust takes off, I've enjoyed playing about with it over the last couple of weeks, although it has made me less happy using the more dynamic languages I normally use to do real work.

[gtani]

What do you mean by subset of scala, is it an absolute prohibition ("It will be flagged, you must be able to justify why you used this") on features used like the Google C++ style guide, or maybe the Levels in Cay Horstmann's book (which I think Odersky came up with), L1-3, A1-3, or something else?