Hacker News new | ask | show | jobs
by the_duke 2539 days ago
There already are a lot of projects in this space.

While better performance is always great, most are already plenty fast for the majority of use cases.

The main power comes from the multitude of inputs and outputs. Vector has a lot of catching up to do there. But if they manage to offer a noteworthy performance gain... one more is always a good thing.

PS: the Logstash numbers seem suspiciously low. I'd bet it's some JVM config issue. Logstash can come to a crawl if it does not have enough memory.
2 comments
otterley 2538 days ago
It's also worth taking into account the size of the software and its relative CPU utilization. Log shippers do require CPU cycles and memory that would otherwise be available to run the other workloads on the host.

As for the multitudes of inputs/outputs, covering the 95% most-used sources and sinks is a great starting point. I think Vector got that list right in this case.

link
leetbulb 2538 days ago
Yep, I push about 50-100MB/s through a single instance of Logstash (Redis (list) -> S3). That configuration is not in the benchmark table, but surely it's more demanding than TCP -> Blackhole, TCP -> TCP, etc.

Regardless, Vector looks very nice and I'll be testing it out :)

link
the_duke 2538 days ago
They are using the default config with 1GB memory. Sadly that's absolutely nothing for Logstash.

Reported an issue in their test harness.

link
otterley 2538 days ago
IMO that's totally fair -- 1GB is more than adequate to run a log forwarder. The fact that Logstash can't perform well under such conditions is worthy of consideration when selecting a solution.
link
the_duke 2538 days ago
It's perfectly valid to mention the higher memory requirements as a considerable drawback.

But this is far from the possible performance that Logstash offers, so the comparison is at least misleading.

Also, there is no information about the memory usage of the other contestants, which probably don't have memory limits if they don't run on the JVM.

I say all this as someone who stopped using Logstash, in part due to the high requirements - but benchmarks should strive for a fair comparison.

link
leetbulb 2538 days ago
I just checked mine and it's running with -Xms1g -Xmx1g and humming away with no problem. I wonder if it's something else?
link