[critium]

I would add to the list;

* If you have transactions, keep your transactions to a single machine/instance/db as much as possible. Muti-machine or software transactions are the LAST solution you should try.

* Pay attention to payload sizes. Make sure you dont come close to saturating the network. Which leads to weird "app is slow" problems.

* Design for testability and diagnosability of production systems. If this is java, use JMX extensions EVERYWHERE for EVERYTHING.

* Time (and timing) is your enemy. Make it your friend.

EDIT: Side note. IMO, JMX extensions are one of the most under-appreciated things that java and jvm devs have but keep forgetting about but its so powerful.

[AdieuToLogic]

  IMO, JMX extensions are one of the most
  under-appreciated things that java and
  jvm devs have but keep forgetting about
  but its so powerful.

This is an excellent point which I cannot agree with more. When doing distributed systems using JVM's, I almost always reach for the excellent Metrics[0] library. It provides substantial functionality "out of the box" (gauges, timers, histograms, etc.) as well as exposing each metric to JMX. It also integrates with external measuring servers, such as Graphite[1], though that's not germane to this post.

0 - http://metrics.dropwizard.io/3.1.0/

1 - http://graphiteapp.org/

[twic]

I'd hesitate to call Metrics 'excellent'. It does quite a bit of processing on numbers before it emits them - averaging over timespans and so on - so if you're feeding them into something like Graphite which does further processing, you're averaging twice, and getting the wrong numbers.

If you're just going to print the numbers out and look at them, it's very handy. But if you want to handle them further, which you really do, I'd avoid it. Put individual measurements in your logs, then scrape them out with Logstash or something and send them to Graphite for aggregation.

[AdieuToLogic]

Use of Metrics functionality is going to vary based on need. So if a system is using something such as Graphite, then the measurements sent should reflect that. This is independent of a particular library IMHO.

So when you say:

  ... if you're feeding them into something
  like Graphite which does further processing,
  you're averaging twice, and getting the
  wrong numbers.

This would be an issue with sending Histogram, Meter, and/or Timer values, but I'd categorize that as a design defect. Using Metrics to send Gauge and Counter data to a Graphite-esque system shouldn't be a problem at all. For systems which do not have a metrics aggregator or when some need to be exposed to external agents (such as client systems), using the types provided by Metrics can be very useful.

As for using log files to "scrape out" system metrics, I realize this is a common practice yet is one which I am fundamentally against. What I've observed when taking the "put it in the logs and we'll parse it out later" tactic is that a system ends up with sustained performance degradation due to high log output as well ops as having to incorporate yet-another system just to deal with profuse logging.

Treating system metrics as a first-class concern has shown itself to be very beneficial in my experience. Persisting them is a desirable thing, agreed, and a feature point fairly easily met when metrics are addressed separately from logging.

[neeleshs]

I agree. I'm a fan of pull model for metrics, where app logs very granular numbers and whoever wants metrics at whatever granularity will pull from these logs into splunk/tsdb etc. Let the app be simple in emitting metrics

[neeleshs]

Beware of memory leaks in histograms!

[AdieuToLogic]

Are you referencing the ThreadLocal[0] issue? If so, it has been committed to master (as mentioned in the comments).

0 - https://github.com/dropwizard/metrics/issues/742