[jbister]

I think it's a little bit beyond "half" baked, but I built a thing I call Logsuck last year: https://github.com/jackbister/logsuck

The idea is to have a free Splunk alternative which you can set up with just one binary. I use Splunk at work and love it, but it just doesn't seem like a product for solo developers (I can't even find a pricing page on splunk.com), and the primary free alternative, the ELK stack, seems a bit complicated to set up.

I am sure that I'll never be competitive with Splunk or Elastic in terms of features or scalability but I'm trying to build something that is at least useful for my own projects.

I built it in Go and use SQLite with the FTS (https://sqlite.org/fts3.html) extension to store the log events in a way where they can be searched quickly.

[remram]

I've been looking into making something like that myself. My beef with current platforms is that they are good for extracting metrics and routing logs to their correct team (by service/deployment/...) but not so much for analysis.

When there's a problem in my system, I want to bring up multiple views, manually annotate, highlight the new debug log statements I just added, mark events as good/bad from complicated rules, etc.

None of the current systems (except maybe Grafana+Loki) really do this, they focus on ahead-of-time ingestion rules and tags. I would love a desktop or web tool that can do that, working from a log file that I can easily extract from my log platform.

[jbister]

I think I know what you mean! It does feel like there's something missing in Splunk when it comes to analyzing issues.

When there's a problem at work it's usually easy to see that something is wrong on a dashboard, but when I want to drill down into it I usually end up with ten tabs of different searches and Grafana dashboards that I'm trying to correlate between manually.

I haven't fully formed any ideas about how to tackle this since there are still fairly basic features missing in Logsuck that I need to work on first but it's definitely something I'll be thinking more about in the future!

[egeozcan]

Wow, I just commented this: https://news.ycombinator.com/item?id=25702294

Maybe I can add yours a JSON view and call it done :)

[jbister]

Interesting! I have it in the backlog that I want to support structured logging via JSON at some point. It's pretty far down the list right now though since I personally haven't used structured logging very much.

I added an issue about it: https://github.com/JackBister/logsuck/issues/7 - if you want to chip in with any comments or even help out with implementing it it'd be much welcomed!

[mamcx]

My main problem making full use of structured logging is the lack of good visualization tool (ie: something that is not a big cluster of stuff :) ). So this look very nice.

Maybe you can combine https://vector.dev that is actually a good shipper and transformation of logs and focus in the visualization and integration? Anyway I hope yours get very good. Will be nice if the whole setup is just a simple go exe and that is all!

[boundlessdreamz]

Vector looks amazing.

Wasn't really happy with any of the existing solutions and ability to collect and export both metrics and logs is just brilliant.

[albertgoeswoof]

This looks great! ELK is overkill for simple apps, sometimes all you want is to feed the logs from a few services into one dashboard.

Datadog is really great for this but can be expensive

[jbister]

Glad you like it! Completely agree about the overkill-ness of ELK.

I haven't had a chance to try Datadog yet but it looks great, the one complaint I've had with Splunk (or at least the way we use it at work) is that it's still "Just logs" and sometimes you need more than that to investigate issues while Datadog seems to cover more areas of monitoring in one package.

[dastx]

Is the aim to have this production ready as an alternative to ELK/Splunk? Or is it more of a tool for local development?

[jbister]

Yes, the goal is that it should be usable in production as well. The limitation it will probably have compared to ELK/Splunk is that since there's only a single recipient node and it uses SQLite for storage, the upper limit to how quickly it can ingest logs will probably be lower compared to ELK/Splunk. So Logsuck may not ever be suitable for large deployments that are logging huge amounts for sustained periods of time.

I haven't really tested what the limits are yet so unfortunately I can't put a concrete number on it, but my vision is that Logsuck should at the very least work for a small-medium sized company with "normal" logging needs.

The current state is that it's got quite a few bugs and is lacking many features you would expect from an ELK/Splunk alternative but the basic concept is working. If what you need is a GUI for grepping across logs spread out on multiple servers it can be used for that today, but it's not doing much beyond that yet.

[ramoz]

This is cool, do you have any usage detail and/or production nuances?

[jbister]

Currently it's not used in production anywhere, I mostly use it to view my own log files when I'm working on some hobby project locally, so I'm sure there are a few issues that would show up if it was used more heavily.

I do have a list of known issues on a private Trello board, I should probably move those to GitHub so that they are at least visible to people, I'll take care of that right away.

There is some info in the readme on how to get started, if you want to try it out and need any help beyond that let me know!