[sandy_coyote]

Yes BUT it needs tuning. Splunk is complicated and takes continuous maintenance to optimize speed.

I work as a Splunk integrator and here's what I often see:

1. Customer installs Splunk with a qualified Splunk or third-party architect team. The deployment works well.

2. Customer adds infrastructure to the deployment. Splunk slows down. License costs go up.

3. Customer chooses between outside help or DIY. DIY rarely works.

4. Customer now needs outside help. Now Splunk is very slow and expensive, and now it will cost a lot to tune it.

Splunk, the company, is in a tough spot for several reasons: rotating c-level cast, unpopular changes to license model, bad acquisitions. The product is still best in class but tough to keep optimized.

[kennend3]

So basically what you are saying is this.

A firm with a competent IT team is unable to get splunk to work and only "outside help" can make the product work?

Given splunks license costs are tied to data ingested, how do you integrate new infrastructure to the deployment and not have license costs go up?

Way to sell us on Splunk?

[bak3y]

Anecdotal - I took over a small, ill maintained Splunk installation at $JOB-2 and reworked it following Splunks current best-practices and it ran like a top as of when I left that place. Having done that process I'm fully convinced that if you're going to run Splunk on-prem you need a dedicated sysadmin for it that knows Splunk's stack. And that kind of person isn't cheap to hire or keep in that role.

[kennend3]

we had an on-prem splunk implementation and it was SOO SLOW.. it was built/managed by splunk and its consultants.

We finally got rid of it a few years later, but for the entire time we had it, it was a constant "round hole square peg" problems. Each time the consultants assured us Splunk could do what we needed, each time it could not.

[chillfox]

I wonder if Splunk has a QA problem with their consultants or if there are certain edge cases they simply don't do well with.

Just that it looks like most people here had a good experience and we had a bad one for some reason.

[bak3y]

Just coming back around to this, we also used Splunk consultants for their SIEM solution and the first one we got wasn't very good, but the second was amazing (I wish we could have hired her directly).

The guy we had help us tune our clusters after I rebuilt them all was also very good. Fortunately I'd done most everything by the books and we overkilled the nodes with hardware (we had some older hypervisor nodes lying around I stole for Splunk).