[_8j50]

Splunk is the best at what it does with no close competition.

I've been looking into Cribl and it seems their product has surpassed their competition as well but not in search, more in data summarization and log reduction, possibly before you ship it off to a more proper place like Splunk.

Splunk's cost makes it inaccessible to most people or companies. I mean, I work in infosec and I highly caution against Splunk because it is so amazing you will hate anything else but in security you need tons of otherwise rubbish data collected centrally sometimes and it will force you into a corner where you will say you can't afford to store that log you really should be storing. Better a crappy tool that can be used to find the logs you need than a nice tool that can only retain so much.

Cribl is supposed to help people reduce what they put i Splunk so they can keep using Splunk, it would have been nice if they partnered instead.

Graylog is another nice tool I like that is somewhat but only slightly similar to Cribl that was founded by a former Splunker out of frustration.

[chillfox]

Is Splunk fast now?

Last time I used it was almost a decade ago and it was rubbish, queries took 10-40 minutes to complete.

[flounder3]

It has always been the Cadillac of search, and moreso with unstructured indexing (e.g. key collisions with different data structures. Foo = string vs foo = integer vs foo = array).

Your queries or infrastructure were not optimized. It’s very fast when optimized.

[chillfox]

Interesting,

It was Splunk managed and configured, so I would have thought it optimized, but I guess they made more money from it not being optimized.

If I remember right then we were throwing about 200+ GB at it a day.

[flounder3]

Splunk worked with us to optimize our configs but we always managed it ourselves.

[chillfox]

Nope, sports betting. I was on the operations side of things and Splunk was something the corporate side organised and championed, but it just couldn't be used to troubleshoot issues in the time frames we needed.

[_8j50]

Very very fast, I can do an all time search on terabytes of data in seconds.

But you have to learn to use it, if you don't give it an index and a sourcetype that will slow it down, and like ES leading wildcards slow things down. The fastest searches are simple terms like a word or an IP.

[chillfox]

Interesting,

From the general responses it sounds like we got unlucky with a dud implementation.

[mattnewton]

40 minutes sounds exceptionally bad, but 5-10 minutes with splunk was totally common when I worked at Apple almost a decade ago, and I could never figure out why because I only ever used it for O(grep on a log file on disk) level operations. I was probably holding it wrong or maybe the infra team had misconfigured it, idk.

[flounder3]

I personally brought Splunk to Apple in 2010 alongside a small handful of people (Hi, Sean and Ariel!). There is a massive difference between real-time searches where latency beyond a few seconds is unacceptable, and historical searches which can take a bit longer. I can assure you that it did its job spectacularly, much to my chagrin that there are few competitors to this day.

[mattnewton]

Cool! That makes sense. I only really used it as part of the (now defunct I think) "orchard" internal hosting platform that was very much beta when I was using it, for tiny internal apps running on like 4 instances at most, and missed being able to just grep log files; my wild, uneducated guess from what you said is that there was some kind of pooling of our meager logs with other people's from orchard, or we were otherwise off the happy path.

[flounder3]

I was part of Orchard and miss it dearly. It had a lot of potential but it was launched as a proof of concept built on pooled resources freed up from optimizing legacy workloads (namely, moving Siri from VMware to Mesos).

It never got the love it deserved and I could absolutely believe that its Splunk cluster suffered as a result. RIP

[mattnewton]

100% agree, the idea of an internal Heroku was a great one, it just didn't seem to work with how Apple was designed organizationally or something and seemed under resourced.

[jitl]

We recently transitioned to it at Notion and it’s been very fast, outperforming the previous log vendor substantially while offering better search and UX. If you used the on-prem version, the cloud version is quite a different experience.

[chillfox]

I don't know which version we used, just that it was managed and configured by Splunk.

We were only sending a small subset of our logs to it so about 200+ GB a day. Our Linux box with spinning disks could grep the full set of logs much faster than querying Splunk, so I don't think anyone really used it.

[throwbigdata]

Yeah but you had clue. Those who don’t use Splunk.

[greggsy]

No offence, but that sounds like lazy query design, poor architecture, or both.

[chillfox]

Maybe we had a bad consultant, I don't know. Splunk were the ones managing and configuring it.

[sandy_coyote]

Yes BUT it needs tuning. Splunk is complicated and takes continuous maintenance to optimize speed.

I work as a Splunk integrator and here's what I often see:

1. Customer installs Splunk with a qualified Splunk or third-party architect team. The deployment works well.

2. Customer adds infrastructure to the deployment. Splunk slows down. License costs go up.

3. Customer chooses between outside help or DIY. DIY rarely works.

4. Customer now needs outside help. Now Splunk is very slow and expensive, and now it will cost a lot to tune it.

Splunk, the company, is in a tough spot for several reasons: rotating c-level cast, unpopular changes to license model, bad acquisitions. The product is still best in class but tough to keep optimized.

[kennend3]

So basically what you are saying is this.

A firm with a competent IT team is unable to get splunk to work and only "outside help" can make the product work?

Given splunks license costs are tied to data ingested, how do you integrate new infrastructure to the deployment and not have license costs go up?

Way to sell us on Splunk?

[bak3y]

Anecdotal - I took over a small, ill maintained Splunk installation at $JOB-2 and reworked it following Splunks current best-practices and it ran like a top as of when I left that place. Having done that process I'm fully convinced that if you're going to run Splunk on-prem you need a dedicated sysadmin for it that knows Splunk's stack. And that kind of person isn't cheap to hire or keep in that role.

[kennend3]

we had an on-prem splunk implementation and it was SOO SLOW.. it was built/managed by splunk and its consultants.

We finally got rid of it a few years later, but for the entire time we had it, it was a constant "round hole square peg" problems. Each time the consultants assured us Splunk could do what we needed, each time it could not.

[chillfox]

I wonder if Splunk has a QA problem with their consultants or if there are certain edge cases they simply don't do well with.

Just that it looks like most people here had a good experience and we had a bad one for some reason.

[HyperSane]

On modern NVMe storage it is INCREDIBLY fast.

[devchix]

> Splunk is the best at what it does with no close competition.

I'm with you. Splunk core - the indexing, automatic parsing, HA architecture, is unsurpassed. You can rebuild/duplicate parts of it but it's not going to come close to what Splunk can do, effortlessly, out of the box. I'm frustrated at the crud that Splunk has acquired which doesn't solve their customer's core problems. Splunk isn't well-rep in the network space. In my past I've worked for a huge tech company that was the darling of its day and Splunk business trajectory reminds me of that; we're within the start of the descent.

I read through the complaints in this thread, how it's slow, behemoth, hard to manage, copmlexities grow ... I've never experienced this problem. I've built and managed 3 Splunk clustered installations, in the 10sTB/day, and I will never use anything else. Sadly, that makes me only able to work for people able to afford the license :nervous laugh: So if you're made of money and want black car white glove data service, buy Splunk and hire people like me.

[gorbachev]

Humio is faster and cheaper than Splunk and while I haven't compared the two products feature by feature, does pretty much the same.

As an end user having used both to manage logs on a few dozen distributed applications I would never choose Splunk over Humio.

[achillean]

There's also Gravwell (https://www.gravwell.io) that competes head-on with Splunk and doesn't punish you for storing/ indexing more data. I'm on their board and knew the founders before I joined so I'm a bit biased but it's basically what if you wrote Splunk from scratch using modern tech.