[dsacco]

I second ELK, and I even stronger-second Splunk being overpriced (with the caveat that if you do deploy it, I think it's the best option, just not really worth it).

I think it's really important to internalize the idea that there is no Platonic ideal of a logging solution. It's a fundamentally frustrating manifestation of entropy that you're going to wrestle with, but it's a really necessary goal to work towards long term. Sort of a "the first step is admitting powerlessness" kind of deal.