Hacker News new | ask | show | jobs
by esafak 706 days ago
It sounds like you are doing ETL on your logs. Most people want to search them when something goes wrong, which means indexing.
1 comments

No, what I'm doing is analysis on logs. That could be as simple as "find me the first N occurrences of this pattern" (which you might call search) but includes things like "compute the distribution of request latencies for requests affected by a certain bug" or "find all the tenants impacted by a certain bug, whose signature may be complex and span multiple services across a long timescale".

Good luck doing that in a timely manner with Kibana. Indexed search is completely useless in this case, and it solves a problem (retrieval latency) I don't (and, I claim, you don't) have.

EDIT: another way to look at this is the companies I've worked at where I've been able to actually do detailed analysis on the logs (they were stored sensibly such that I could run mapreduce jobs over them) I never reached a point where a problem was unsolvable. These days where we're often stuck with a restrictive "logs search solution as a service" I often run into situations where the answer simply isn't obtainable. Which situation is better for customers? I guess cynically you could say being unable to get to the bottom of an issue keeps me timeboxed and focused on feature development instead of fixing bugs.. I don't think anyone but the most craven get-rich-quick money grubber would actually believe that's better though.