[jsmeaton]

I think you make a good point that logs should be transformed closer to the source. I work, primarily, with applications provided by a vendor, with very unstructured log data. Transforming (Grok) these logs is an absolute must, we couldn't look at something that didn't allow transformation. That said, maybe we should be looking at something closer to the source before handing it off to a central location. Are you aware of agent-like daemons that do transformation before handoff?

[seanp2k2]

Structured logs are awesome and a great idea. For the next few decades while standards come and go and everyone gets it all implemented across the board, yes it sucks to write grok patterns for the flavor of the week, but once you do it a few times, it takes maybe a few hours of work to get some app cluster with moderately logging flowing into ES with all the right types and all the edge cases accounted for. From there, ELK is such a Swiss Army knife that it's worth the trouble, since then it's e.g. trivial to fire PagerDuty alerts off if you hit some exception-level log lines, or post metrics about your logs, or put them on some queue to flow into some big data pipeline thing.

[b0ti]

You might want to consider NXLog if you need to do transformation at the source. For us this was an explicit design goal. Moreover it is also lightweight and a lot of people use it in place of other fat and bulky solutions, quite popular with ELK users.

[nathwill]

We use heka and love it