[radq]

Relevant comment: https://news.ycombinator.com/item?id=6799328

" CloudFlare generates 50gb/s of logs globally and have handled collecting this volume in two ways. Historically the logs are sent to a local syslog-ng through the use of a PIPE and the forwarded to central logger. This can be done with nginx with no patches by just treating the PIPE as file. Just make sure you do a little buffering inside nginx.

access_log /dev/nginx_access log_format_name buffer=64k flush=10s;

Since this is a pipe there is still some blocking IO, but no worse off then writing to local file."

[otterley]

If the pipe consumer (i.e. your syslog server) stalls, then it will be much worse than writing to a local file, as it could cause a denial of service.

I would still configure nginx to write to a file and have the consumer tail it to avoid this situation. Properly-configured log rotation can keep the file size within reasonable bounds.

[jsmeaton]

Related question - how do you configure logrotate to honour size immediately? In my (limited) experience, logrotate runs once overnight. If you have a log file that expands by 1gb per hour, and you want to rotate it every 500mb, how do you instruct logrotate to handle it?

Do you just crontab logrotate for short intervals, and will that trigger "daily" rotations each time for other configurations?

[otterley]

There's no reason why you can't run logrotate more than once a day. You can run a separate instance every 15 minutes if you like, with a separate configuration file that only rotates the short-lived logs.

[jsmeaton]

Ok that's the (easy) thing I was missing. Instead of using the main logrotate.conf (that includes logrotate.d/*), use a fully separate logrotate.conf. Thanks.

[asdasf]

>If the pipe consumer (i.e. your syslog server) stalls, then it will be much worse than writing to a local file, as it could cause a denial of service.

But how likely is that to happen in any circumstance that doesn't also bring down nginx anyways? Obviously in the case of a large setup like that they don't care, it just gets removed from the pool regardless of why it is failing.

[staunch]

Extremely likely. I've seen the rsyslog daemon die randomly many times, for example. Whereas in 6+ years of using Nginx I've never seen it fall over a single time. Web serving is usually a primary purpose for a server. You do not want your web site to be down just because the machine's syslog daemon died.

[voltagex_]

Isn't this a reason to replace the syslog daemon rather than the webserver?

[staunch]

It's a reason to keep things as loosely coupled as possible. And when I say I've seen rsyslog crash many times, I'm talking about over the course of years and thousands of servers.

[asdasf]

Our notions of "extremely likely" differ then. I have never had syslogd crash. If you have, you should be switching to a different syslogd.