[zak_mc_kracken]

> for the simple reason that Java is designed to be mainstream,

Not really. Java was designed to be practical, and it ended up being mainstream.

Most languages want to become mainstream (including Scala), most of them just fail to achieve that goal for a variety of reasons, often due to their design decisions (but not always).

[vorg]

> Most languages want to become mainstream (including Scala), most of them just fail to achieve that goal for a variety of reasons, often due to their design decisions

A language will often become used in one particular domain, often not the domain it was designed for. Take another JVM language Groovy -- it's used a lot for scripting throwaways used in testing, including build files for Gradle, and was extended with a meta-object protocol for Grails which has risen and fallen in sync with Rails. But then its backers retrofitted it with a static typing system and have tried to promote it for building systems, both on the JVM and Android but virtually no-one's biting.

[bad_user]

I disagree, Java was designed to be marketable to the layman and for a very good reason ...

Software companies can scale either horizontally (e.g. hiring more people) or vertically (hiring the best). Because of the classical bell curve distribution, really good people are hard to find, therefore horizontal scalability is very much preferred (and obviously not everybody can have "the best"). Unfortunately you can't really grow a company both horizontally and vertically. The reason for it is because getting good results from mediocre people requires lots of processes, coordination/synchronization and politics of the sort that doesn't appeal to really good people and that's because really good people have a choice, because of the supply-demand equation which favors them.

In the wild you'll also notice another reason for why horizontal scalability is preferred. Corporate departments and consultancy companies are usually scaled horizontally because they get paid per developer/hour and unfortunately there are only 24 hours in a day. Hence the bigger the team, the bigger the budget. On the other hand small teams happen in startups where you often see 2-3 people doing everything. Small teams also happen in big software companies, when speaking of either privileged seniors or folks that handle real research.

Big teams are getting things done, as Alan Kay was saying, on the same model as the Egyptian pyramids, with thousands of people pushing large bricks around. Small teams on the other hand tend to handle complexity by means of better abstractions. Unfortunately better abstractions are harder to understand, as they are first of all, unfamiliar.

So back to Java ... it's actually a pattern that you tend to notice. Such languages are afraid of doing anything that wasn't proven already in the market. The marketing for such languages also relies on fear and doubt of the alternatives, which Sun took advantage of, appealing to the insecurity of the many. This is one language that refused to implement generics because they were too hard and then when they happened, the implementation chose use-site variance by means of wildcards, with optional usage, possibly the poorest choice they could have made. Generic classes are invariant of course, except for arrays. This is the language that doesn't do operator overloading, as if "BigInteger.add" is more intuitive than a "+", but on the other hand it does have a + for Strings. This is the language in which == always does referential equality, so you've got tools converting every == to .equals because that's what you need most of the time. The language which until recently didn't have anonymous functions because anonymous classes should be enough. This is the language in which everything needs to be in a class, even though the concept of "static method" makes no sense whatsoever. This is the language in which at least 30% of all libraries are doing bytecode manipulation to get around the language's deficiencies. Such languages also avoid metalinguistic abstractions like a plague, such as macros, because OMG, OMG, people aren't smart enough for that (though truth be told, you're expanding your language every time you write a function, but don't tell that to the weak). Also, AOP is a thing that has a name in Java, which is pretty funny, given that in other languages it tends to be effortless function composition.

As an environment it turned out to be great, eventually. If you want another shinning example, take a look at Go. Same story.

In other words Clojure or Scala will never reach the popularity of something like Java. Sorry, people can wish for it, but it will never happen. And that's OK.

[justthistime_]

+1.

If I wanted a worse-is-better language to work with worse-is-better developers, I have plenty of languages to choose from.

Personally, I prefer using a good language. Judging from all the butt-hurt responses some people here think that I'm not allowed to. I wonder why.

[pron]

I disagree. Java's design wasn't timid -- it was radical in its intentional reduction of expressivity compared to the most popular language around, because the much more expressive language was proven to be far too costly to maintain for the duration software has to be maintained.

But you are certainly right that Java was also meant to be familiar and unthreatening. It allowed itself to do that because it realized -- and was proven right -- that extra-linguistic features such as a GC and dynamic linking are essentially the features that contribute to productivity more than abstractions.

> Software companies can scale either horizontally (e.g. hiring more people) or vertically (hiring the best).

While this may be true, it has little to do with Java directly. For years I worked with the best algorithmeticians, physicists and mathematicians (physics simulation, real-time control). We were very much involved with the hard problems we were trying to solve with novel algorithms, and frankly couldn't be bothered with spending precious resources on elegant abstractions. Our problems was one of essential complexity -- not accidental complexity -- so we needed a simple language that can be easily shared and understood by everyone, one that was fast, and one that got the job done.

Believing that better developers opt for more powerful languages is what I call the "Blub developer paradox": the blub developer has only worked on simple, blub, problems (CRUD applications, web etc.), and hence believes his cool abstractions are the mark of the "smart developer". If he'd only look at developers working on seriously hard algorithms, he'd realize that abstractions are secondary, and it is developers above him on the essential complexity spectrum that opt for simpler languages with weaker abstractions.

That better developers choose the more powerfully-abstracting languages is completely and utterly false. The best developers I've ever known -- those who really came up with radical, game-changing solutions that truly advanced the state-of-the-art -- were all C and Java people, and I don't think it is a coincidence. Their mind is focused on the algorithm rather than on alluring, but ultimately distracting abstractions.

However, saying that only people who work on boring problems have the free mental resources to spend on nifty abstractions is as valid a generalization as the one you've made. There are many reasons to choose a particular programming language.

Personally, though, I believe that while the choice of the programming language matters, it matter so much less than design, choice of algorithms and availability of other tools. I no longer equate software with the code it's written in. A running program has many factors affecting its quality and cost, and the code used to write it is just one of them. Still, if I need to write a web app, I'd choose Clojure over Java every time. If I need to write a military command-and-control app, or a power-station control app? I'd probably go with Java.

[bad_user]

You're attributing to me and my message things I haven't said, which is a sign of an unproductive conversation.

> For years I worked with the best algorithmeticians, physicists and mathematicians (physics simulation, real-time control)

If we are talking about experience, I've built an RTB system with soft real-time requirements handling tens of thousands of transactions per second and in which Scala was awesome because we used better abstractions for handling multi-threading, concurrency and processing of huge amounts of data in real-time.

Right now I'm working on a project for E.On that monitors and controls power plants. We are talking about real-time processing of signals, modeling of state machines that evolve according to those signals and machine learning for finding the optimal dispatch algorithm and for better fault detection. Scala has been awesome because functional programming along with its very static type system allowed us to better handle the event-based nature of the system and the business logic which is like a freaking fractal of complexity.

I'm now moving on to another project from the health-care industry that handles, you probably guessed it, monitoring of patients by analyzing signals coming from medical devices. Also in Scala, though there's lots of flexibility required in configuring the system and I might propose Clojure for some parts because read-eval.

Thinking that you're the only one that interacted with "the best algorithmeticians, physicists and mathematicians" or that your problems are more interesting than what other people are doing is snobbery. I never said that the best people choose different languages, all I said is that the best people choose better abstractions and that some languages are meant for building better abstractions. Speaking of which, for "blub CRUD applications" I would never choose something like Clojure or Scala, simply because for those types of apps PHP or Ruby have got that covered.

[pron]

I don't think I'm the only one etc.. It's just so happens that all the best people I've known to work on algorithms preferred languages with minimal abstractions. I very well acknowledge that others, who are no less talented, might prefer more abstractions.

However, I am saying that the assumption that better developers invariably prefer more abstractions and more powerful languages is absolutely wrong.

Personally, I've had a terrible experience with Scala, found its abstractions distracting, its complexity hindering readability, its DSLism obstructing code sharing and staff migration among various teams, and the grapple with the tool stack a horrible waste of time -- all with little benefit and lots of harm. I know that my experience isn't universal, but it's far from unique, either.

Clojure is a completely different story, partly because its designers are not purists and not interested (at least not as a primary concern) in PL research, but even with Clojure I don't think we have enough data to conclude that its cleverness (which, thankfully, is much lower than Scala's[1]) is a net gain for large projects that require maintenance for a decade or more by large teams.

All in all, I've seen that a solid, fast runtime with a good compilation and linking story, excellent profilers and deep monitoring (all provided by the JVM) yield much bigger gains than most new languages. Given that those extra-linguistic features are free and switching a language is extremely costly, I'm always hesitant about whether it's worth it. Staying on the JVM reduces the switching costs, and good interoperation with Java -- like Clojure has -- reduces it further, so I find the risk to be worth it for some less-risky projects.

[1]: Then again, so is every other language in history.