[steveBK123]

I was at a shop that got heavily integrated into Splunk for security use cases and then entered a split brain mode of 'well if you need observability we already have Splunk' but also 'hey stop doing so much observability, this thing is expensive!'.

So for 5 years time we used it for observability, we were only half-integrated and also trying to get off of it. Great stuff.

[dharmab]

Worked on a piece of software which suffered from years of this split brain. It had some logging and some metrics, but the team was told to be economical about observability. This resulted in the software having many blind spots which led to production issues that had to be manually reproduced. When I become responsible for the software I personally overhauled the logging and the team had to work together to rebuild the metrics functionality.

[steveBK123]

this is an area that gets very political with architects, managers and other non-coders having too much of a say

a lot of paralysis on the app dev side as the status quo is easier than fighting for a sensible outcome

its also something that yes, benefits stakeholders... but only on a 2nd/3rd order effect of outage avoidance & remediation.. so theres not a huge reward for doing it really really well in many shops

[0xBDB]

Pretty sure every Splunk customer has that split brain. This thing's great, what can we quit sending to it?