[agerlic]

Great post and diagrams. How do you manage/centralize logs ?

[steveklabnik]

Thank you, they took forever to draw... My handwriting is terrible.

We wrote about logging here: http://blog.balancedpayments.com/status-page/

  > We already log these to a centralized server using RSYSLOG, so I already had
  > a data source to draw from. Next, I went and brewed a fresh pot of coffee and
  > bestowed it upon bninja for his prescient work in building our log parser,
  > Slurp. We wrote a quick Slurp script that read the HTTP status code from each
  > request and then fed them into Graphite buckets. Each bucket was based on
  > service name (DASHBOARD, API, JS) and then response code family (2xx, 3xx,
  > 4xx, 5xx, and a special case timeout for slow requests).

If infrastructure stuff is interesting to you, you may want to check out https://github.com/balanced/balanced-infra . If there's interest, I might blog about it in the future.

[rafekett]

what do you do when rsyslog drops log events?

[steveklabnik]

I am not an ops person, but my understanding is that rsyslog does 'store and forward', which would allow replaying of the log if something was dropped: http://www.rsyslog.com/storing-and-forwarding-remote-message...