[lmm]

Heh, I was wondering when you'd show up :)

The study sounds interesting, though there are obvious risks to trusting self-reports, but I think you're taking a broader meaning of "familiarity" than they're using - I read the result as "programmers are likely to use a language that they know when starting a project" and I don't think you can generalize to "programmers are more likely to use a new language that's similar to a language they know" (apologies if there's something in the video that's not in the text, can't watch videos here). Certainly past a certain point the library ecosystem and pool of familiar programmers become self-sustaining, and I can well believe that such things would be the biggest factors in any given project's choice - but that can't possibly explain how a language gets to that point in the first place.

For many of us the distinguishing characteristic of Java was that it was heavily marketed, often to managers; in the early days it felt like a lot of Java users were doing so under orders. (With 1.5 and the introduction of generics the language became a lot more programmer-friendly and I think at that point it developed a lot more grassroots popularity). Being less cynical I think Java did offer something unique in the form of crossplatform multithreading with a well-defined memory model - it's easy to forget how valuable that seemed, particularly as the world moves away from conventional threading. Possibly the combination of memory-safety and claims of C-like performance was new as well? I'm speculating here as I'm not so familiar with the less popular alternatives, but I think it's not simply a case of Java being less weird or more blue-collar than Self or Modula. The popularity of Python (with weird syntax and modula-like modules) and Javascript (with a Self-like inheritance model) show that that level of weirdness is no barrier if a language has a compelling advantage.

(Likewise Go, I think - it's taking programmers largely from Python/Ruby land where it can claim a USP in the form of performance (while retaining good async support and some level of conciseness). And again, marketing seems to be somewhat involved - so often when I ask "Why did you choose Go over OCaml?" the answer is "What's OCaml?".)

I think the clearest strike against the blue-collar approach is Perl. Perl took an extremely "pragmatic" attitude to language design, adopting many "most bang-for-the-buck features" (deemphasising consistency or underlying coherence), and put a lot of emphasis on tooling and libraries with CPAN, PerlUnit and the like. And for a time it was very successful - but ultimately it became a language that was very hard to evolve, whereas other languages (in particular Python and Ruby) were able to catch up on the tooling side and their coherence gave them an advantage in the long-term. I think Kotlin is making the same... mistake would be too strong a word, Perl is a success story by many measures, but I think Kotlin is taking a very short-term strategy; it feels like every time someone figures out a clever thing you can do with implicits in Scala it gets added to Kotlin as a language feature. Everything about it feels very ad-hoc, particularly in comparison to Ceylon. So I think that while Kotlin may be a very effective language for the short term it's going to be a very brittle one that will struggle to adapt in the future.

I guess that's the closest thing to a counterexample to my original claim - a lot of people did move from Perl to Python with no USP beyond being clearer and smoother. FWIW I think the issues with Perl were deeper than those with Scala; there is some ad-hoc junk in the language (structural types should never have been added, nor should Dynamic, async/await aren't worth it...) but it feels quite peripheral, I think the core still fits together nicely. But hey, if those issues do end up forming enough of a toehold for Ceylon to take over, I'm fine with that too.

[pron]

You won't find much argument from me here, but I think I need to clarify my point. I'm not saying that blue-collar is "better" or even has better chances for adoption[1] (though I believe it does), but that when you decide to go non-blue-collar, how exactly do you do that? For example, you mention "deemphasising consistency or underlying coherence" as a bad thing. But how bad is it? We know memory safety is important. Do we know sound type systems are important to the same extent? My important meta-point was that when designing a new language for the industry, you should at least do so to solve a problem that you believe (preferably with empirical evidence) causes real pain in the real world. That's how Java, Erlang, Clojure, Rust and even Kotlin were designed (Kotlin is different because it's decided to solve a less-painful problem but for much less cost, but the same thinking is there). Are crude type systems a real problem? I don't know, but I certainly have never seen any evidence to suggest that is the case. So if you set out on this path, at least conduct a survey or research (or even your own rich experience) to convince yourself that your solving a major problem. Why? Because unlike many other tools, new languages are very expensive to adopt. The higher the cost -- the bigger the benefit you must provide.

Having said that, as you correctly note there are other paths to adoption, such as being the only language on a popular platform (JavaScript, Objective-C, Swift), or being a scripting language. Those seem to have their own rules.

[1] Although Java wasn't more marketed than other heavily-marketed alternatives at the time, and Python and JavaScript both have their own peculiar adoption stories, with JavaScript still trying to look similar to its "sister-language" Java, and then being the only choice on a popular platform, and Python taking a long time for decent adoption that even today is far from C/Java/C# levels of popularity, and either case being, or at least starting out as scripting languages for small programs -- not as languages for "serious" systems.

[lmm]

I'm not sure the evidence is any stronger. I mean, you can point to some bugs and say they were caused by a lack of memory safety, I can point to some bugs and say they were due to a poor type system.

Certainly there were problems I faced in day-to-day Java programming that were visibly due to a crude type system - NPEs, the awkwardness of working with checked exceptions (I do think Ceylon has missed a trick by not using union types to interoperate with Java checked exceptions), SQL injection vulnerabilities caused by not having a taint checker. Other problems I recognised as being due to crude typing in retrospect - the fact that missing transaction annotations could failed silently in confusing ways ( http://thecodelesscode.com/case/211 ), the ease of forgetting to close files, the way all the ways of doing async were either invisible or painfully verbose. Ultimately, everything that we resorted to (inherently non refactor-safe) annotations for. (For what it's worth, I originally picked up Scala based on my experience with the JSR308 checkers framework - the functionality was valuable but too many tools didn't interoperate with it properly, so I looked for a language that had the same thing built-in).

Any individual problem from that list you can solve at the language level - some of them have been solved in Java already, and more in the likes of Kotlin. But a powerful type system is a tool that, once it's built into the language, lets you solve them all even if you didn't think of them first. (Macros have the same kind of power, but maintainability of macros is a real problem). I don't think we'll find an example of a powerful type system being better at a specific feature any more than we'd find an example of a generic data structure being better than a specific one; the value comes in being able to do many things the same way.

Making a language that can be evolved effectively is certainly a real problem - again Perl is the clearest example, but I've also heard complaints about how long it's taken Java to implement features that have been agreed on for years (lambdas, modularization). I don't know whether it's the problem King was intending to solve with Ceylon (though I do think Rust represents a very deliberate effort at the same thing) and I don't think it's easy to prove that you have solved it short of waiting a few decades - I certainly don't have any hard evidence about what factors make a language able to evolve in the long run. Nor would it be a compelling marketing story even if you could prove it. But I think it's still worth working on.

[pron]

I really like your answer. I think it's very important to start with actual problems and then work your way towards a solution rather than vice-versa, because I've always been more troubled by all the bad things people do with rich types[1] rather than notice the importance of problems they do solve. On the other hand, any powerful general tool built into the language can be more trouble than it's worth. Macros are a good examples, but types can be worse. Macros are local to an expression, but types carry the magic with the variable. Sometimes types just carry restrictions, which is OK, but when traits/typeclasses/implicits are involved, they carry actual "bad" magic. So while other specific solutions may take time to implement, codebases live for a long time, so I'd rather wait and suffer some inconvenience, than have people turn a codebase into an unmaintainable, magical mess (which is what I've heard large companies using Scala say it's done to them). Linear types and intersection types are indeed less prone for abuse, and are mostly "restriction-only", so they seem safer, and are less likely to be abused for data types rather than cross-cutting concerns. Another approach -- my favorite -- is exactly Checker. I agree that at this point in time its UX is pretty bad, but it really doesn't have to be. It's mostly a result of the U-of-W putting more effort into the actual mechanics rather than the UX (which, IMO, is a mistake, but they're a university and naturally care more about research...). The Java annotation-processing mechanism generally works very well with IDEs.

Another question is, how bad those problems you mentioned really are. Concurrency bugs are terrible, and simple data typing bugs are usually a non-issue, but tainting/resource-use (i.e. solvable with intersection/linear types respectively) may be bad enough. I really wish someone would conduct an empirical research comprising a taxonomy of bugs, their frequency, and their cost. Such data would be infinitely more useful for language design in the industry than, say, all research going into session types (not trying to pick on session types research, but I hope you catch my meaning when I speak of actual applied utility).

BTW, I think any interesting idea is worth working on. The problem is that a language is a very expensive tool -- and a risky one -- so when it comes to industrial, rather than academic, use a lot of care should be taken when picking one, so it's better to err on the side of caution.

[1]: See Scala's collections as an extreme example. Is it really necessary to go through all that trouble of unreadable code just to save us a cast here and there that has never really bothered anyone?

[lmm]

Hmm, the reason I find types clearer than macros is that they do apply everywhere, so there's consistency across a codebase. <- will always mean a call to flatMap - which might do something different in one place or another, but only in the same way as a normal polymorphic method call - whereas different macros might use it to mean unrelated things. I think IDEs handle them better too - you can refactor in the normal way, you get types on mouseover. Last time I tried to use Checker it was similar - the IDE has good support for running the checkers, but it can't and won't apply the right annotations when extracting a method, or show you the checker-inferred types of local variables.

I do think implicit conversions allow too much; IMO we need typeclasses and extension methods. So my ideal design for a language would include implicit parameters but not implicit conversions, and instead have dedicated support for extension methods. That said we do need to be able to use typeclasses without disrupting method application syntax (i.e. support the use case that means Spray prefers the implicit-conversion-based "magnet pattern" rather than a typeclass).

More generally I think there will be languages that allow too much and languages that ban too much, and we need languages on both sides of the line to allow us to converge on something better. Most places I've worked have had a strong code review/pull request culture which may have coloured my experience - I just haven't seen the Scala maintainability problems you talk about.

(If I thought the complexity of collections was necessary to avoid casts then I would defend it - casts can fail and so it's very nice to have a codebase where they're banned entirely (enforced with wartremover). But I don't think it is - FWIW Odersky's asked for proposals for scala collections in version 3, and I think there's a substantial push towards at least some simplification - perhaps replacing CanBuildFrom with a 1-parameter typeclass that would retain most of the power and simplify the code. The showing-off cases like BitSet#map would require one more method call, but no casts)