Java at first is great. Once you get in to large and complex loads, the GC tuning becomes load specific. While tuning GC can be fun (for the first dozen times), the bigger problem happens when your load shifts and your GC tuning becomes subtly wrong.
Also even IF you are tuned properly, there is a world of 99-percentile you'll never get to.
Additionally, the difference between openJDK and Oracle JDK becomes something admins learn to hate you for. Complex shell scripts to invoke java, and lots of standard unix tools just don't work super well. eg: pgrep and pkill. you can tweak it, but it takes a little while to learn the many tips and tricks.
It's not all bad, most Java programs are deployed in a static-all-batteries-included fashion, so you rarely worry about system-installed library versions. So that makes deployment a little less hassle. You never have to recompile for cross platform. The profiling tooling and other stuff is pretty good, and the more you're willing to pay the better the tools get.
You are wrong about never getting to the 99th. I did not work on this service at Twitter, but many services at Twitter are built on the JVM, and if this service is like most there, then it probably has SLA targets that are sub-millisecond at the 99th and sub 10ms at four 9's. The only caveat to this is that it has to write to disk, which may keep it from achieving truly high performance, but this is not the fault of the JVM. I worked on one service at Twitter that had to use microseconds on its dashboards just to see any variation at all throughout the day.
The point is you cant use binary name to make your way around anymore. The binary name is 'java'. Standard unix tools just don't work. Yes there are work arounds, but over time things end up being just a little more complex than they should be.
Which means if someone is comparing zookeeper and etcd, well etcd wins major points for being unixy and easy to deploy. Copy 1 binary, done. ZK loses major points here. Gotta make sure the JVM is installed, but do you need the OpenJDK or the Oracle one? if the latter, well apt-get and yum are less helpful.
It's all just little globs of annoying details that add up to be a small pain. Nothing horrible, but if you could make a choice to avoid that, why not?
Basically I guess what I'm saying is there is probably a market for replacing all the Javay distsys stuff with Rust/Go versions. I mean look at etcd!
> Basically I guess what I'm saying is there is probably a market for replacing all the Javay distsys stuff with Rust/Go versions. I mean look at etcd!
Or in the other direction. If your main system runs on the JVM and your sysadmins are used to the JVM tools then having a piece of infrastructure that's just another .jar is wonderful, and C/Rust/Go/Ruby/etc. infrastructure elicits groans. Mixing platforms will always be harder than a common platform. So the infrastructure market depends on where you think the future of applications is.
For good C, which is hard to write - for me and other mortal humans.
On the other hand, as a mortal, I can write a-grade-above-code-that-an-idiot-would-write-just code in Java at about 10 times the speed I can write dire-useless-risible C code.
The comparison is pointless though, good modern languages like Rust and Julia are developing and LLVM is enabling further development.
Also even IF you are tuned properly, there is a world of 99-percentile you'll never get to.
Additionally, the difference between openJDK and Oracle JDK becomes something admins learn to hate you for. Complex shell scripts to invoke java, and lots of standard unix tools just don't work super well. eg: pgrep and pkill. you can tweak it, but it takes a little while to learn the many tips and tricks.
It's not all bad, most Java programs are deployed in a static-all-batteries-included fashion, so you rarely worry about system-installed library versions. So that makes deployment a little less hassle. You never have to recompile for cross platform. The profiling tooling and other stuff is pretty good, and the more you're willing to pay the better the tools get.