[andygrunwald]

We at trivago had a similar problem. For this we created Gollum: - http://tech.trivago.com/2015/06/22/gollum/ - https://github.com/trivago/gollum

We use it heavily to stream any Kind of Data into Kafka: Access and errorlogs, Application Logs, etc. Did you consider it as well?

[whost49]

No, we did not consider Gollum--it definitely looks like a possible solution and one we might have considered. I think the name of the project makes it hard to find, unfortunately.