Sounds crazy, but Datadog. I’ve been hammering their product teams for years with specific use cases for the sole purpose of replacing Splunk. They recently migrated search technologies and are rapidly closing the gap. Plus, their exclusion features are instant and fantastic, and their C-suite replies to me when I escalate.
Elasticsearch simply couldn’t handle key collisions. We have hundreds of various apps across 5-10 different languages and frameworks where a key name may be reused as either a string or a hash or an integer or an array. If we can’t freeform search (which Splunk is EXCELLENT at), we just need to be able to transform the data beforehand. Datadog plans to do so with their recent acquisition of Vector.
Sad. Splunk should be more fantastic. They have done the heavy lifting of taking streams of data at high volume, which should be the basis to build a log search product, metrics And alerting, and observability.
Instead, each of these systems have their own collectors and correlating from one to the other is hard. A canonical log line is so much more valuable than a metric collected every 60 seconds, and the former can derive the latter: https://stripe.com/blog/canonical-log-lines
I built a PCI compliance solution for a customer back in 2008 for ~$200k all-in when the closest competitor's bid was five times that. The product was amazing at runtime but of course had some idiosyncrasies in how it was configured and whatnot. I've been a user (only) of Splunk heavily ever since and just last year got pulled into a project to migrate a huge install to a cloud platform. It felt like I got into a time machine...there were seemingly zero administrative or architectural improvements to make the product more manageable or supportable in the 10+ years since I had last looked at it from an ops perspective.
I'm sure that's not 100% true but it felt like it. Trying to build Splunk on top of a modern IaC deployment methodology is a huuuuge lift.
Would love to know more about specific use-cases you've been talking to Datadog about. I'm starting a company (log-store.com) that I pitch to people as 75% of the features of Splunk at 50% the price. Right now that 75% is probably more like 25%, and the 50% is _actually_ 0%... it's FREE! Any and all feedback is greatly appreciated!
This is the question. If you’re looking for APM well you’ve got great options but for those using Splunk in the security space (SIEM & SOAR) you’re screwed.
There’s no better SIEM alternative that deals with logs at scale.
Splunk recently screwed a friends Fortune 50 company. They didn't pay a bill on time (renewal negotiations) and Splunk without even contacting them just left all the logs from one of their instances on the floor. They lost everything for literally an entire country.
Lol, this is exactly what I was referring to! We were negotiating a massive renewal (50x budget IN EARNEST!) and out of nowhere they threatened to cut us off with less than 24 hours notice because we were a week behind their schedule, despite signed agreements. Thankfully our VAR (a longtime partner) jumped in and cut a PO to vouch for us and our word.
NONE of our other vendors have EVER done that shit in my entire career. EVER. My word means nothing to them; they act like a pure private equity player now.
Lol, I could have written exactly the same comments you did about Splunk. I'm the one who decided to start using splunk for our company 8 years ago, we're working on migrating away and I will never under any circumstances consider recommending splunk for any future companies I work for.
I can truly empathize. I’ve never wasted so much of my time on a vendor. It’s infuriating and certainly not worth the personal stress and agony. It shouldn’t be this hard!
Can echo the licensing sentiments across 2 companies now. It's a shame because I grew to really like SPL and how Splunk handled web based distributed search. What are you moving towards now, if I may ask?
It's pretty ridiculous this can happen. With building Matano (https://github.com/matanolabs/matano), an open source security lake, one of our core decisions was to store all data in S3 in Apache Iceberg tables (an open table format that lets you query data from any supporting tool i.e Spark, Presto, Athena, Snowflake, more). This let's you own your data without it being held hostage on some vendors instances in a proprietary format.
Wait, what? When a license expires my understanding is indexing continues, you just can't search, you can get a key to unlock search by contacting Splunk. I assume you mean the SaaS Splunk Cloud, then, because the renewal period would be far in advance of the license expiring, Splunk doesn't just "stop". That behavior makes no sense and would lose of a lot of clients, you can't just backfill data gaps.
>There’s no better SIEM alternative that deals with logs at scale.
I think folks that use Splunk for basic search just don't fully comprehend how capable the product is for hunt-type operations when someone fluent in SPL is at the helm.
I can't agree more. I've used every main 'competitor' now and nothing can compare to splunk for hunting across massive logging pools. It genuinely feels like magic with advanced SPL and solid regex.
My frustrations with Splunk have been around their certification and training changes over the years. Used to be able to get a solid tool certificate and decent training materials all for free. It only hurts Splunk though as less people have experience with the tool it lessens their advantage. Makes me disappointed as I really do like the tools itself but literally everything else is terrible. I'd much rather deal with Elastic or go open source with Security Onion.
Ex-splunker here. I just started working at FeatureBase and would say, if your data is in Kafka, FeatureBase might be something to consider. It’s a crazy fast binary index built on Roaring bitmaps.
Elasticsearch simply couldn’t handle key collisions. We have hundreds of various apps across 5-10 different languages and frameworks where a key name may be reused as either a string or a hash or an integer or an array. If we can’t freeform search (which Splunk is EXCELLENT at), we just need to be able to transform the data beforehand. Datadog plans to do so with their recent acquisition of Vector.