| > I've got a scala app that regularly consumes about 48G of ram, and I'm very happy with the response times during heavy loads like this, but the P99.5 is abysmal because of garbage collection. I've tried tuning it, but it doesn't seem like anything I do helps. When tuning a prod Scala deployment a while back, I encountered a nasty "pregnant pause" every once in a while similar to what you have experienced. So, below are what I used in annotated form and adapted for the deployment you describe (the max heap setting). Some of these may be completely obvious to you (or others reading), yet are included for completeness. -server
This one should be obvious :-)
-Xmx54G
Memory is cheap, so give the JVM 54 gig so that GC
isn't forced to run when your system is in the
steady-state of 48G heap utilization.
-XX:PermSize=128m -XX:MaxPermSize=1024M
Ditto on the cheapness of memory.
-Xss1M
A stack size of 1 meg seems a bit much, but does
allow for recursive algorithms to operate with
impunity.
-XX:ReservedCodeCacheSize=128m
This one was needed for Scala. It likely should
be specified with a high value since it limits the
JIT's code cache.
-XX:+DoEscapeAnalysis
A nice way to releave some heap pressure[1].
-XX:+UseCodeCacheFlushing
Should the ReservedCodeCacheSize be exceeded, this
lets the JIT continue to do its thing in an LRU
type of fashion (I believe).
-XX:+UseParallelGC
This one is the most impactful one of all. A lot
of people will say "use UseConcMarkSweepGC!" They
are wrong for high volume server deployments. The
concurrent mark and sweep algorithm caused massive
"pregnant pauses" in prod for me! The Parallel GC
algorithm performs much better under load and
doesn't cause the VM to sit-and-spin for 20+
seconds.
-XX:+UseCondCardMark
Another tweak which had a major performance boost
for me[2].
-XX:+UseNUMA
If your servers are NUMA[3] based, then this can
significantly increase performance[4] as well.
HTH1 - http://www.ibm.com/developerworks/java/library/j-jtp09275/in... 2 - https://blogs.oracle.com/dave/entry/false_sharing_induced_by... 3 - https://en.wikipedia.org/wiki/Non-uniform_memory_access 4 - http://jose-manuel.me/2011/06/numa-bb/ EDIT: Inserted newlines to eliminate horizontal scrolling. |