[reacharavindh]

I was dragging my feet to build a log shipper solution. I was going to use Filebeat -> ElasticSearch -> Kibana.

This looks great. My primary attraction is possibly low memory footprint of this program over Filebeat. Secondary attraction is how easy it appears to enable transformations.

Now, if I can make a suggestion for your next/additional project..... A neat system metric collector in Rust that can export to Prometheus with same principles.

Low memory footprint,

Rust,

Single binary,

Customizable with a single config file without spending hours in manuals,

Stdin, Stderr -> transform -> Prometheus.

I’m learning Rust and eventually plan to build such a solution but I think a lot of this project can be repurposed for what I asked much faster than building a new one.

Cheers on this open source project. I will contribute whatever I can. Thanks!!

[lukes386]

Thank you! Very glad it looks useful to you.

It's still slightly rough around the edges, but Vector can actually ingest metrics today in addition to deriving metrics from log events. We have a source component that speaks the statsd protocol which can then feed into our prometheus sink. We're planning to add more metrics-focused sources and sinks in the future (e.g. graphite, datadog, etc), so check back soon!

[wikibob]

Just a question, are you familiar with work that's been done on the OpenCensus Collector and Agent [0]?

There was discussion earlier this year about creating a design doc for OpenCensus to handle logs. I'm not sure if that got finished, or it was sidelined while the OpenCensus & OpenTracing merger was worked on. Both projects will combine under the OpenTelemetry name.

I've been quite happy with the OpenCensus instrumentation SDKs.

I think the Logs & Metrics space is interesting, especially because there is so much overlap, both are just ways of representing data about an event that occurred in the software.

OpenCensus is fairly widely backed: Google, Microsoft, Etsy, Scalyr...[1]

[0] https://github.com/census-instrumentation/opencensus-service...

[1] https://opencensus.io/community/users/

[lukes386]

I've looked into OpenCensus/OpenTracing/OpenTelemetry (and the apparently unaffiliated OpenMetrics?) a bit, but I'm not as familiar as I'd like to be. It does seem like they're focused primarily on application-level instrumentation and the ability to ship that metrics and tracing data to different backends.

Vector's perspective is that your applications and infrastructure are already emitting all kinds of interesting data via logs, metrics, etc, and the primary challenge is to collect, enrich, and manage the storage of that data. We have no plans to integrate Vector into your application or introduce some kind of Vector-specific method of exporting data.

We'll definitely be watching OpenTelemetry as it moves forward and would very much like to be a compatible part of that ecosystem. To the degree that they use common open standards for their communication protocols, that could just fall out naturally.

[navaati]

What you're asking for kinda exist, it's called mtail, check it out !

[lukes386]

We really love the idea of mtail and think it's criminally underused.

A big part of the reason we started building Vector was to integrate that kind of functionality into a larger project, so people wouldn't have to get over the hump of discovering, setting up, and rolling out a whole separate tool.

We're definitely not as mature as mtail yet, but we're working really hard to get there.

[Thaxll]

You better use something reliable and tested like Filebeat, it's written in Go and is fast enough / low memory usage.