[nkozyra]

Does anyone think about doing logging to shared memory / memcached and then committing snapshots to disk at regular intervals via another process/machine?

If you're not all that concerned about consistency, each web server can keep their logs in a segregated memory space and then another process can combine/commit and send a flush command, leaving the primary machines relatively unencumbered.

[evan_miller]

With fsync() taken out of the equation this is essentially how a naive logger operates. The linux page cache plays the role of shared memory and only under rather heavy contention will a write() incur latency.

The trouble outlined in my blog post is that the logging framework was calling fsync() -- that is, specifically asking to flush the page cache all the way to disk.

[jacques_chester]

Have you experimented with a log-structured filesystem? I once saw similar high-variation behaviour in a logging problem on extfs. Switching to NILFS pretty much got rid of variable latency entirely.

[evan_miller]

Not recently, no. I do know that a journaled filesystem can exacerbate this sort of problem as it can make extra work. For example: http://lwn.net/Articles/328363/

In a few cases in the past when dealing with unimportant data I have downgraded to ext2 for a nice performance bump.

[jacques_chester]

Just to make sure we're talking about the same thing, a journalling FS isn't the same thing as a log-structured FS.

The first has a write-ahead log, the latter is basically just a log. So immediately writing to disk is relatively simple.

[evan_miller]

Yes. I haven't done anything with a log-structured FS. I have only played around with conventional filesystems.

[spydum]

Isn't this essentially syslog-ng?

[nkozyra]

Does syslog-ng use shared memory? I thought it communicated via open socket network channels.

[spydum]

apps can log using the syslog primitives. I do not believe the require network sockets. What syslog-ng decides to do with it (flush to disk, or push over network socket), is no different than the proposal of shared mem to another process (which then has to make the same decision).