[jacques_chester]

Have you experimented with a log-structured filesystem? I once saw similar high-variation behaviour in a logging problem on extfs. Switching to NILFS pretty much got rid of variable latency entirely.

[evan_miller]

Not recently, no. I do know that a journaled filesystem can exacerbate this sort of problem as it can make extra work. For example: http://lwn.net/Articles/328363/

In a few cases in the past when dealing with unimportant data I have downgraded to ext2 for a nice performance bump.

[jacques_chester]

Just to make sure we're talking about the same thing, a journalling FS isn't the same thing as a log-structured FS.

The first has a write-ahead log, the latter is basically just a log. So immediately writing to disk is relatively simple.

[evan_miller]

Yes. I haven't done anything with a log-structured FS. I have only played around with conventional filesystems.