[alxnlssn]

I ended up setting up one machine with a 80GB tmpfs mount for the graphite data, and then rsync it to disk every hour. That allows carbon-cache to keep up, but I'm not happy with the setup.

Whisper is terrible for spinning disks.

[dothebart]

if you use collectd to feed values into graphite, you've got the advantage of its bulk writes. This article describes how its solved for rrd only collectd installations: https://collectd.org/wiki/index.php/Inside_the_RRDtool_plugi... but the effect also becomes visible if you use it to send values to graphite.

Its also good in reducing the amount of data you need to send to graphite.

[vidarh]

How do you figure? Unless recent versions of Whisper have been totally rewritten, whisper writes each metric to a separate file. Submit hundreds of metric per vm/server every 10 seconds, and you get ridiculous amounts of tiny writes (e.g. 4 byte writes) fenced by redundant seek()'s and a number of other syscalls, no matter how much you batch up stuff before sending it to statsd.

[Shish2k]

I came up with the same workaround myself; then switched to influxdb, and my monitoring server's io-wait is still at 0% even with twice as many metrics coming in :)