|
|
|
|
|
|
by tmzt
1966 days ago
|
|
|
You mentioned Loki in your post. I evaluated it for our company and was reasonably impressed with the simplicity of setup and efficient storage. Where it failed us was the difficulty searching by customer identifiers or other "high cardinality" labels, or full-text. There's a longstanding issue [1] on the Github for this. Are you doing anything to improve log search versus an Elasticsearch cluster, for instance? More broadly, how are you contributing to the upstream projects? [1] https://github.com/grafana/loki/issues?page=7&q=is%3Aissue+i... |
|
|
> Are you doing anything to improve log search versus an Elasticsearch cluster, for instance?
No. You're right, Loki is not designed for building up an index for full-text search. The premise here is that you won't typically need that and that in exchange for not having to build up that index, you get other advantages (such as being able to rely on an object store for both payload and index data!). If, on the other hand, in special situations, you need to "grep-search" your logs, this is absolutely doable with Loki! Loki does not neglect this use case. The opposite is true; everyone is already excited about the performance characteristics that Loki already has today when it comes to ad-hoc processing full text. For example, see https://twitter.com/Kuqd/status/1336722211604996098 and definitely have a look at https://grafana.com/blog/2020/12/08/how-to-create-fast-queri.... I'm sure Cyril is happy to answer your questions, too!
> how are you contributing to the upstream projects?
We're reporting issues and try to contribute as much as we can! Of course, this effort has only started. So far, we've contributed to Loki's Fluentd plugin (https://github.com/grafana/loki/pulls?q=is%3Apr+author%3Ajge...), and our testing efforts have helped reveal edge cases; see for example https://github.com/grafana/loki/issues/2124 and https://github.com/grafana/loki/issues/3085.
We're excited to substantially contribute to both Loki and Cortex in the future!