Hacker News new | ask | show | jobs
by flounder3 1350 days ago
Splunk, as a company, is a shell of its former self. All they care about is pimping themselves out to maximize profits to an extreme that only Dilbert can relate to, even at the expense of destroying a long term professional relationship over trivial matters. They are more than happy to kill a deal over a 5% disagreement rather than understand the needs of a Fortune 500 customer and negotiate.

They are mad because Cribl is good at transforming data before it ingested by Splunk, so as to reduce the amount of data that is indexed. Period.

Splunk ONLY RECENTLY released “Ingest Actions” to filter data post-ingest (to avoid indexing) for their SaaS product — something that has always been a mainstay of their on-premise “Enterprise” product. Their ONLY suggestion to filter data that we didn’t care to index in early 2021? Cribl. There’s literally no other reason for us to use Cribl.

I’ve been paying for Splunk since 2008 and can’t wait to get away from them. Their sales teams have decayed into unethical slimebags and I am trying everything in my power to not renew our contracts with them. This just sealed the deal.

Source: I cut checks to Splunk for $x,xxx,xxx yearly

4 comments

I can add from the other side of the fence. I worked for a startup that was acquired by Splunk. They are everything listed here and worse on the inside.

My first few weeks at Splunk were very odd. They try to indoctrinate new hires with a barrage of "A-players" that continuously talked about how awesome Splunk was. Except... When I started Splunk was getting their ass kicked by cloud-first players that had recently come to market. Splunk's monolithic architecture wasn't well suited to be run as SaaS at the time and Splunk was burning cash and losing money on every customer that they suckered into moving away from their perpetual licenses into subscription hell. I left money on the table when I ran out the door less than 6 months later.

I'm curious what Splunk's long game is with this because they just told every F2000 that their bottom line is being chipped away by Cribl and friends. So if I'm an enterprising procurement department I'd be tossing Cribl or Rudderstack or whatever other data transformation preprocessor on the table alongside my renewal. Expand opportunity? If you put your ear to the tracks you can almost hear all of the account managers digging out missed quota excuses.

Splunk isn't innovative and hasn't been for a long time. Most of the employees saw the writing on the wall and went to Snowflake as soon as the opportunity presented itself. Splunk tried to capitalize on the security market by, basically, double charging customers for ES. Instead of delivering value it seems to be Splunk is just looking for ways to squeeze a few last drops of lemonade.

Sounds like a good move... From their about page:

    Splunkers have received over 1,020 patents to date
that tells me everything I need to know.
I am one of those people. There was a bonus for every patent granted. They were telling us that we need to big patent arsenal to fend off against IBM. It turned out that Splunk is IBM now.
“You either die a hero or you live long enough to see yourself become the villain”
I am against software patents and choose to ignore all pleadings from my employers regarding patent filings. IMO, the bonuses (~$1-2K) are not worth going against my views.

You could have made the same choice, but did not.

Your last sentence is rather dismissive and seems unnecessary to make your point. You're assuming that they share your views on software patents. They might not. Or at that time not realise the issue with software patents in the first place.
Hear hear. I know what you mean, I've lived through the same in a different Fortune 100.
Interesting, I helped manage a splunk install at a fortune 200 about a decade ago. At the time the recommendation was to use syslog-ng to filter incoming logs before indexing. I just heard of cribl 2 weeks ago because the fortune 20 I currently work for is planning on switching to it. I didn't realize it was a massive shift like that, I just thought it was the corporation switching things just because they do that sometimes.
There have been a few other recommendations over the years, including putting a separate tier of forwarders first in line to perform transforms and such. There were always plenty of options for on-prem/DIY/Enterprise especially when using syslog instead of directly via HEC.

Their SaaS offering used to have said inline tier called IDM (Inputs Data Manager) where we were directed to configure filters during our POC… a key requirement for moving from Enterprise to SaaS because conf files aren’t managed the same. One month (to the day!) after we moved, they randomly decided to migrate us to a new “Victoria experience” where that tier suddenly disappeared without explanation. We filed support tickets asking 1) what happened? and 2) how do we filter things out now? and were directed to hire professional services because that was outside the scope of standard support!

The whole point of moving to SaaS was to not have to babysit our own clusters (small shop at the time), so spinning up a ton of infra in front of the freshly greenlit SaaS setup would have negated the productivity gains and financial pivot.

Ultimately, the entropy of hundreds of applications logging in disparate formats and namespaces outweighed our ability to sanitize each app within a reasonable amount of time, leading to unwanted data being indexed, ergo overages. Overages that our sales engineer originally assured us we could address by filtering things out with the snap of a finger. Bait and switch.

Ingest Actions were not available at the time, and were not functional (even in beta) until 10 months later.

No comment about the company, but want to make clear as a buyer you understand the procurement and legal parts i.e. MFN or MFC.

If they do discount, even 5%, then it ripples across their accounts as a legal matter, esp at your scale. I was a buyer for some big companies, 8 digit, and the procurement office would only do a deal with MFN/MFC clause. They would also audit the supplier from time to time.

I totally understand that ripple effect and am very familiar with Most Favored X when it comes to unit pricing of a tangible good (e.g. xx,xxx physical servers with a particular SKU), but in this case we were talking about a SaaS product where overages were disputed. Nearly every vendor would jump at the chance to discount additional commitments or support at the ‘expense’ of waiving some past overages.
Thanks for the response, been there on the overages per SAAS’s. Now running a startup, they scare me even more.
Last used splunk around 2010, and we ran a bunch of scripts to truncate and reformat logs before they got anywhere near our splunk data load license
What are you planning to move to?
Sounds crazy, but Datadog. I’ve been hammering their product teams for years with specific use cases for the sole purpose of replacing Splunk. They recently migrated search technologies and are rapidly closing the gap. Plus, their exclusion features are instant and fantastic, and their C-suite replies to me when I escalate.

Elasticsearch simply couldn’t handle key collisions. We have hundreds of various apps across 5-10 different languages and frameworks where a key name may be reused as either a string or a hash or an integer or an array. If we can’t freeform search (which Splunk is EXCELLENT at), we just need to be able to transform the data beforehand. Datadog plans to do so with their recent acquisition of Vector.

Sad. Splunk should be more fantastic. They have done the heavy lifting of taking streams of data at high volume, which should be the basis to build a log search product, metrics And alerting, and observability.

Instead, each of these systems have their own collectors and correlating from one to the other is hard. A canonical log line is so much more valuable than a metric collected every 60 seconds, and the former can derive the latter: https://stripe.com/blog/canonical-log-lines

Splunk should have been the lynchpin.

Absolutely agree. It’s tragic.

Shoving SignalFx down our mouths and trying to get us to create “metric” indices was the straw that broke my back.

I built a PCI compliance solution for a customer back in 2008 for ~$200k all-in when the closest competitor's bid was five times that. The product was amazing at runtime but of course had some idiosyncrasies in how it was configured and whatnot. I've been a user (only) of Splunk heavily ever since and just last year got pulled into a project to migrate a huge install to a cloud platform. It felt like I got into a time machine...there were seemingly zero administrative or architectural improvements to make the product more manageable or supportable in the 10+ years since I had last looked at it from an ops perspective.

I'm sure that's not 100% true but it felt like it. Trying to build Splunk on top of a modern IaC deployment methodology is a huuuuge lift.

Yeah, team decided K8s for Splunk would be too much work and ended up needing to use vanilla VM’s with block storage on an open stack env on prem.

Pretty lame not cloud native.

Would love to know more about specific use-cases you've been talking to Datadog about. I'm starting a company (log-store.com) that I pitch to people as 75% of the features of Splunk at 50% the price. Right now that 75% is probably more like 25%, and the 50% is _actually_ 0%... it's FREE! Any and all feedback is greatly appreciated!
Sounds crazy indeed. I worked at Splunk for many years and was a DataDog customer later. The costs of either are not something I care to deal with.
You might also want to review NewRelic. Their new plans allow for lots of data ingestion at low price including logs.
Exactly.

This is the question. If you’re looking for APM well you’ve got great options but for those using Splunk in the security space (SIEM & SOAR) you’re screwed.

There’s no better SIEM alternative that deals with logs at scale.

Splunk recently screwed a friends Fortune 50 company. They didn't pay a bill on time (renewal negotiations) and Splunk without even contacting them just left all the logs from one of their instances on the floor. They lost everything for literally an entire country.

I mean EVERYTHING.

Lol, this is exactly what I was referring to! We were negotiating a massive renewal (50x budget IN EARNEST!) and out of nowhere they threatened to cut us off with less than 24 hours notice because we were a week behind their schedule, despite signed agreements. Thankfully our VAR (a longtime partner) jumped in and cut a PO to vouch for us and our word.

NONE of our other vendors have EVER done that shit in my entire career. EVER. My word means nothing to them; they act like a pure private equity player now.

Lol, I could have written exactly the same comments you did about Splunk. I'm the one who decided to start using splunk for our company 8 years ago, we're working on migrating away and I will never under any circumstances consider recommending splunk for any future companies I work for.
I can truly empathize. I’ve never wasted so much of my time on a vendor. It’s infuriating and certainly not worth the personal stress and agony. It shouldn’t be this hard!
Can echo the licensing sentiments across 2 companies now. It's a shame because I grew to really like SPL and how Splunk handled web based distributed search. What are you moving towards now, if I may ask?
It's pretty ridiculous this can happen. With building Matano (https://github.com/matanolabs/matano), an open source security lake, one of our core decisions was to store all data in S3 in Apache Iceberg tables (an open table format that lets you query data from any supporting tool i.e Spark, Presto, Athena, Snowflake, more). This let's you own your data without it being held hostage on some vendors instances in a proprietary format.

-https://github.com/matanolabs/matano

- https://iceberg.apache.org

> They didn't pay a bill on time (renewal negotiations)

I don't side with Splunk on their actions but I can't stand customers who withhold payment as a renewal tactic and think that it's equally wrong.

Wait, what? When a license expires my understanding is indexing continues, you just can't search, you can get a key to unlock search by contacting Splunk. I assume you mean the SaaS Splunk Cloud, then, because the renewal period would be far in advance of the license expiring, Splunk doesn't just "stop". That behavior makes no sense and would lose of a lot of clients, you can't just backfill data gaps.
>There’s no better SIEM alternative that deals with logs at scale.

I think folks that use Splunk for basic search just don't fully comprehend how capable the product is for hunt-type operations when someone fluent in SPL is at the helm.

I can't agree more. I've used every main 'competitor' now and nothing can compare to splunk for hunting across massive logging pools. It genuinely feels like magic with advanced SPL and solid regex.

My frustrations with Splunk have been around their certification and training changes over the years. Used to be able to get a solid tool certificate and decent training materials all for free. It only hurts Splunk though as less people have experience with the tool it lessens their advantage. Makes me disappointed as I really do like the tools itself but literally everything else is terrible. I'd much rather deal with Elastic or go open source with Security Onion.

Ex-splunker here. I just started working at FeatureBase and would say, if your data is in Kafka, FeatureBase might be something to consider. It’s a crazy fast binary index built on Roaring bitmaps.