[38529977thrw]

It would have been worthwhile for them to mention why not use Java. I mean, grep for Java and JVM and that short blog lights up! It was good that they did stress the (very) small team size for the product as well. This was an aesthetic choice and I wish he had stressed that.

Java remains an excellent choice for concurrent systems, with a commonly accepted static type system, and all "java.util.concurrent" and the rest of JVM are native and the language and tools do scale to even monstrous sized "teams".

[aphyr]

I've been a Java programmer for ~18 years! A nontrivial portion of Jepsen is written in Java too. :-)

Many of the points I touched on in the post are direct critiques of Java, but maybe I should be explicit. An obvious factor is size--when I've written the same project in both languages, Clojure usually winds up 5-10 times smaller. That's definitely improving as Java adds streams, lambda expressions, and so on, but it's still a good deal more verbose. That gets in the way of the kind of exploratory programming I do, especially at the REPL, and it's harder for me to read and maintain a giant codebase. I appreciate having fewer nouns, in general: Clojure emphasizes a uniform way of traversing and transforming data structures, and in Java every single class has its own way of representing its data, and many of those ways are bad.

The standard collections library is full of mutability and thread safety pitfalls. Printed representations of datatypes are verbose. It's not particularly good at dealing with highly polymorphic data structure transformation, and it's not a particularly good static type system either--a good part of my Java Brain (TM) is devoted to knowing the mechanisms of erasure, memory layout, and when unsafe casts are actually the right choice. Functions are sort of becoming first-class with method refs and functional interfaces, but it's nowhere near the convenience and flexibility of a Lisp-1. Accessing the compiler at runtime is a pain in the ass. No hygenic macro system makes things like custom iteration or compositional error handling a pain. There's no analogue to Clojure's protocols, which are a fantastic tool for "Hey, compiler, I want a monomorphic-cached type-dispatch polymorphic call site here for a type hierarchy I don't control". Etc etc.

There are things I love about Java. Automated refactoring is easier, and I generally like the level of IDE support. Nearly anything involving primitives, I drop to Java. Ditto, APIs that require annotations. Interface specification is more rigorous. Etc etc. That's why I write in both languages. But most of Jepsen is in Clojure for good reasons. :-)

[38529977thrw]

Thanks for the thoughtful reply. Always interested in your thoughts on software matters.

You know my professional issue with Clojure is that it attracts poseurs. It's that strange spot in PLT that to appreciate it requires sophistication and experience yet barrier to entry (contra say Haskell) is much much lower and does not require the same. So you can get bragging rights without being someone like you who actually understands the cost equations in toto in context of picking languages. This human factor coupled with the technical matter of Clojure not being ideally suited for large scale code / long running / typical IT fubar realities.

You mentioned macros. I can tell you about 'sacred macros' that must not be touched :) Clojure may have addressed software compatibility but it has a human resources compatibility issue.

[Capricorn2481]

> Nearly anything involving primitives, I drop to Java

Can you describe what you mean by this? Does this just mean you're using native Java data types sometimes when speed is a concern? Is there an example somewhere in Jepsen?

[aphyr]

Speaking very loosely, primitives on the JVM are values which are represented directly in memory, instead of as pointers to objects on the heap. Clojure (again, very loosely) generally treats everything as a pointer to a heap object. There is no specialized equivalent for, say, a vector of shorts, or a map where values are floats. The compiler can emit specialized function signatures for... IIRC longs and doubles, but other types (e.g. byte, float) aren't directly accessible--they go through widening conversion. It's also easy for the compiler to quietly fail to recognize it can preserve primitives in some kinds of loops, so you wind up with what Java calls "autoboxing": wrapping a primitive in a corresponding Object type.

Here's a recent example of some code in a hot path inside Elle, one of Jepsen's safety checkers. It does a lot in primitives, using packed structures and bitmasks to avoid pointer chasing.

https://github.com/jepsen-io/elle/blob/main/src/elle/BFSPath...

There was actually a Clojure version of this earlier that got pretty close perf-wise, but I wound up dropping to Java for it instead:

https://github.com/jepsen-io/elle/blob/913cbff5ebb19ba850c0a...

[Capricorn2481]

How often is this necessary? I haven't been able to make an example of Java code performing faster than Clojure. I tried to make the java equivalent of this in Clojure

``` (defn sum-clojure [size] (reduce + 0 (range 0 size)))

(sum-clojure 100000000) ```

Despite the fact that Clojure primitives are boxed, a manually constructed long array that was summed together in Java was much slower. Why is that?

[aphyr]

Necessary depends on your use case! I spend a lot of time waiting on analyses, and the more operations I can test, the more bugs I can find. I probably invest more time in performance optimization than most people.

Regarding your specific example, uh, I don't know how you measured, but that feels... off. Here:

``` package scratch;

public class Sum { public static long sum(long[] longs) { long sum = 0; for (int i = 0; i < longs.length; i++) { sum += longs[i]; } return sum; } } ```

``` (require '[criterium.core :refer [quick-bench bench]]) (def longs (long-array (range 100000000))) (defn sum-clojure [size] (reduce + 0 (range 0 size))) (import 'scratch.Sum) ```

Summing an array of 10^8 longs like you suggested takes ~120 ms on my machine.

``` user=> (quick-bench (Sum/sum longs)) Evaluation count : 6 in 6 samples of 1 calls. Execution time mean : 119.804166 ms Execution time std-deviation : 9.124709 ms Execution time lower quantile : 115.154905 ms ( 2.5%) Execution time upper quantile : 135.597497 ms (97.5%) Overhead used : 20.384654 ns

Found 1 outliers in 6 samples (16.6667 %) low-severe 1 (16.6667 %) Variance from outliers : 15.4197 % Variance is moderately inflated by outliers ```

Your Clojure function, which sums a lazy range, takes ~4.7 seconds--about 40x slower.

``` user=> (quick-bench (sum-clojure 100000000)) Evaluation count : 6 in 6 samples of 1 calls. Execution time mean : 4.739575 sec Execution time std-deviation : 346.895187 ms Execution time lower quantile : 4.557732 sec ( 2.5%) Execution time upper quantile : 5.324739 sec (97.5%) Overhead used : 20.384654 ns

Found 1 outliers in 6 samples (16.6667 %) low-severe 1 (16.6667 %) Variance from outliers : 15.3163 % Variance is moderately inflated by outliers ```

An idiomatic Clojure reduction over the same array of longs, just so we're measuring apples to apples, takes about 12 seconds--about 100x slower.

``` user=> (reduce + longs) 4999999950000000 user=> (quick-bench (reduce + longs)) Evaluation count : 6 in 6 samples of 1 calls. Execution time mean : 12.037821 sec Execution time std-deviation : 433.661044 ms Execution time lower quantile : 11.790637 sec ( 2.5%) Execution time upper quantile : 12.779262 sec (97.5%) Overhead used : 20.384654 ns

Found 1 outliers in 6 samples (16.6667 %) low-severe 1 (16.6667 %) Variance from outliers : 13.8889 % Variance is moderately inflated by outliers ```

Incidentally, this is one of the reasons I wrote `loopr` (https://aphyr.com/posts/360-loopr-a-loop-reduction-macro-for...). One of the iteration tactics it can compile to is iteration over arrays. Nowhere near Java--I think it's probably still boxing a fair bit, and I'd need to disassemble it to see--but it's still 10x faster than the standard reduce here. ~10x slower than Java.

``` user=> (quick-bench (loopr [sum 0] [x ^"[J" longs :via :array] (recur (+ sum x)))) Evaluation count : 6 in 6 samples of 1 calls. Execution time mean : 1.259095 sec Execution time std-deviation : 3.876650 ms Execution time lower quantile : 1.256692 sec ( 2.5%) Execution time upper quantile : 1.265747 sec (97.5%) Overhead used : 20.321072 ns

Found 1 outliers in 6 samples (16.6667 %) low-severe 1 (16.6667 %) Variance from outliers : 13.8889 % Variance is moderately inflated by outliers ```

[Capricorn2481]

Edit: I wonder if there's a caching thing going on. I am having wildly different performance depending on what number I put in. Does criterium affect this by isolating something?

I read a little more about this before you replied, but it seems like we're doing something different or there's a JVM difference that is bigger than expected.

- The clojure function takes me less than a second.

- (reduce + (long-array 100000000)) also takes less than a second.

- From what I understand, the compiler may have a special rule around using reduce and + that uses unboxed numbers by default

- Isn't it a big "if" that you might have a big array of primitive longs in Clojure? I can see how if you've already gone through a whole range of numbers and unboxed them then the java function would be fast. But it seems like it'd be easier to use + (again, this is running a lot faster on my machine for whatever reason).

I'm new to all of this so just being academic and trying to figure out what's going on