[Papric0re]

[Offtopic (a bit)] Lots of you are talking about metric monitoring. But do you have recommendations when it comes to (basic) security Monitoring? I would usually go for the Elastic-Stack for that purpose, especially because Kibana offers lots of features for security monitoring. But I feel like these stacks are so big and bloated. I basically need something to monitor network traffic (Flows and off-Database retention of PCAPs) and save some security logs (I'm not intending on alerting based on logs, just for retention). But being able to have a network overview, insight into current connections (including history) is a very useful thing. Can anybody recommend something, that's maybe a bit lighter than an entire Elastic-Stack?

[floren]

I think Gravwell (https://gravwell.io) might be what you're looking for--but I work for Gravwell so I may be biased! If I can be forgiven a short sales pitch, we've built a format-agnostic storage & querying system that easily handles raw packets, Netflow records (v5, v9, and IPFIX), collectd data, Windows event logs, and more. You can see some screenshots at https://www.gravwell.io/technology

We have a free tier which allows 2GB of data ingest per day (paid licenses are unlimited) which should be more than enough for capturing logs and flows. The resources needed to run Gravwell basically scale with how much data you put into it, but it's a lot quicker to install and set up than something like Elastic, in our opinion (https://www.gravwell.io/blog/gravwell-installed-in-2-minutes)

Edit: it's currently a bit roll-your-own, but we're really close to releasing Gravwell 4.0 which enables pre-packaged "kits" containing dashboards, queries, etc. for a variety of data types (Netflow, CoreDNS logs, and so on)

[iudqnolq]

When you say

> Gravwell is developed and maintained by engineers expert in security and obsessed with high performance. Therefore our codebase is 100% proprietary and does not rely on open source software. We love open source, but we love our customers and their peace of mind a lot more!

does that mean you've even rolled your own webserver? Programming language?

[floren]

That's... not good copy. I think it must have been written long ago. We use open-source libraries (with compatible licenses, of course) and even maintain our own set of open-source code (https://github.com/gravwell). I'll talk to the guys who maintain the website and get that fixed. Thanks for pointing it out!

Edit: We've had lots of people assume we use Elastic under the hood, so I wonder if that was just a (poorly-worded) attempt to indicate that our core storage and querying code is custom rather than some existing open-source solution.

[Papric0re]

Maybe you should just wipe that paragraph completely. I get that investors like to see that you are using proprietary code, but I wouldn't expect you to be faster with that. Especially when running against Elastic, which has over 1.400 contributors currently. But you don't necessarily need to. You can get me with being focused on the right thing and not bloating your software. Lot of big projects start to loose focus and start doing everything, hence become worse doing their main job.

Especially when it comes to security, I'd like to see the lowest complexity possible. Harden your software instead of feature-fu around. That would be a good USP (I've got the feeling that no vendor has realized this so far - but customers neither did).

[Papric0re]

I don't mind you giving a small sales pitch...and maybe your product is indeed what I'm searching for. But your pricing model is instantly putting me off. Same as with Splunk, you end up with not being able to predict your cost and paying way too much. Tell me when you fix that and I might be interested ;)

Edit: Sorry, I was misreading your comment. Premium is unlimited...I will look into it, thanks. :)

[floren]

Yep, paying customers are licensed by the node rather than by the gigabyte (as Splunk does it), and you're really only limited by your hardware at that point. You might be surprised at how much you can accomplish on the free license, though--there are several small businesses using it to monitor their networks because 2GB/day will hold a pretty hefty amount of Netflow, collectd, Zeek, and syslog records.

[mavam]

(Disclaimer: CEO & founder of Tenzir)

We at Tenzir are developing VAST for this purpose: https://github.com/tenzir/vast. It's still very early stage, but if you're up for trying something new, a lean and modern C++ architecture, BSD-license open-source style, you may want to give it a spin. The docs are over at https://docs.tenzir.com/vast.

It supports full PCAP, NetFlow, and logs from major security tools. There is CLI and Python bindings. The Apache Arrow bridge offer a high-bandwidth output path into other downstream analytics tools.

[retzkek]

Maybe Loki [1] meets your needs? It lacks the analytics abilities of Elastic (e.g. what's the average response time [2]) but is much simpler to setup and use for jog aggregations, and has a pretty powerful query language for digging through and graphing log statistics (e.g. how many errors have been logged per hour). It's mainly being developed by Grafana Labs, so there's great integration in Grafana.

[1] https://grafana.com/oss/loki/

[2] I'd argue this sort of thing should be published as a metric anyways so you don't have to pull it out of the logs

[Papric0re]

Tanks for the tip. But I would still need some storage and Data Shipper, right? Or is Loki also taking care of storage?

[tilolebo]

Promtail is the official log shipper for Loki, but you can also use others. See https://github.com/grafana/loki/blob/master/docs/sources/cli...

As for storage, the default is BoltDB for indexes and local file system for the data, but you can also use popular cloud solutions like DynamoDB, etc. AFAIK BoltDB is automatically installed when you install Loki.

The only possible pain point I see for you is that Loki is tailored for Kubernetes. It is totally possible to use it without running a K8 cluster, but you lose some features.

[user5994461]

If you want to do logs, you can use graylog or kibana, both using elasticsearch for storage. This allows to find what was connecting where at some point in time (HTTP request logs and database connection logs).

If you want to graph connections from service to service in real time. I've actually never found anything that was capable of doing that, not even paid software.

[tilolebo]

Grafana together with Loki would be a good match for you