[bak3y]

Anecdotal - I took over a small, ill maintained Splunk installation at $JOB-2 and reworked it following Splunks current best-practices and it ran like a top as of when I left that place. Having done that process I'm fully convinced that if you're going to run Splunk on-prem you need a dedicated sysadmin for it that knows Splunk's stack. And that kind of person isn't cheap to hire or keep in that role.

[kennend3]

we had an on-prem splunk implementation and it was SOO SLOW.. it was built/managed by splunk and its consultants.

We finally got rid of it a few years later, but for the entire time we had it, it was a constant "round hole square peg" problems. Each time the consultants assured us Splunk could do what we needed, each time it could not.

[chillfox]

I wonder if Splunk has a QA problem with their consultants or if there are certain edge cases they simply don't do well with.

Just that it looks like most people here had a good experience and we had a bad one for some reason.

[bak3y]

Just coming back around to this, we also used Splunk consultants for their SIEM solution and the first one we got wasn't very good, but the second was amazing (I wish we could have hired her directly).

The guy we had help us tune our clusters after I rebuilt them all was also very good. Fortunately I'd done most everything by the books and we overkilled the nodes with hardware (we had some older hypervisor nodes lying around I stole for Splunk).