[mikeshi42]

Luckily ClickHouse and serious throughput are pretty synonymous. Internally we're at 100+PB of telemetry stored in our own monitoring system.

Vector supports directly writing into ClickHouse - several companies use this at scale (iirc Anthropic does exactly this, they spoke about this recently at our user conference).

Please give it a try and let us know how it goes! Happy to help :)

[atombender]

Thanks! Very familiar with ClickHouse, but can logs then be ingested into CH without going through HyperDX? Doesn't HyperDX require a specific schema that the Vector pipeline would have to adapt the payloads to?

[mikeshi42]

Nope! We're virtually schema agnostic, you can map your custom schema to observability concepts (ex. the SQL expression for TraceID, either a column or a full function/expression will work).

We don't have any lock in to our ingestion pipeline or schema. Of course we optimize a lot for the OTel path, but it works perfectly fine without it too.

[atombender]

Very happy with CH, but I'm sadly disappointed with HyperDX:

* Not possible to reorder columns?

* Not possible to wrap cells?

* The doesn't seem to be a concept of "field popularity" as Kibana has (where you can also "pin" fields)?

* The log view's chart is very simplistic. No breakdown. Time selector is very primitive. Look at Google Cloud's log view if you want to see something good here.

* No field value autocompletion when using the query builder?

* Live view is annoying as hell. It scrolls to the top every few seconds even when there is no need data.

* Chart view is nearly useless. I tried to create a chart showing two time series calculations, average and P95 of a metric, and it doesn't draw it correctly, and the series get messed up, and overall I think it's not usable. Happy to explain further, but even a cursory test should reproduce this. Fortunately Grafana can access CH data and do a much better job here.

* The drill-down sidebar doesn't seem to have any idea what's important or not, so some fields I will never want to filter on are high up in the list and others are further down. I can't rearrange the fields?

* Lots of other nuisances.

Overall I found it much, much weaker than Kibana for both logs and charts, and Kibana is already pretty atrocious at both, so.

I can probably live with it because I'm desperate to replace Elastic with CH, but I think I will kiss the functionality of the Kibana UI.

[mikeshi42]

This is really helpful feedback, a few notes:

- Reorder columns: You should be able to do so by modifying the order of the SELECT statement at the top.

- Wrapping: totally agree - this is something we're adding in, it's on the near term todos.

- Pin fields: you're totally right, you can pin values but that doesn't prioritize the filter in general. I'm getting this one in the queue.

- Time picker: I hear you with Google Cloud's time picker, they did indeed make a really nice one. Though I'm curious to hear more about the breakdown, is it wanting to customize how that chart is grouped beyond log level?

- Autocomplete: We had a regression if you're on v2.0.0 which is what I suspect you're hitting. If you're on v2.0.1 lucene should have field value autocomplete. We don't have it yet for SQL.

- Live View: Can you clarify what is scrolling? Or is this about disabling live?

- Chart View: I just tried really quick and it seems okay to me, do you mind sharing more details?

- Sidebar: Is this related to field popularity to pin fields or something else?

This feedback is all super helpful - you can probably tell we're still early in building out the perfect experience. I'd love it if we could dive in deeper either on our discord (https://hyperdx.io/discord) or email: [e-mail]. Either way this has been exceptionally helpful :)

[atombender]

Thanks for the reply. I'll probably show up on Discord eventually!

- Reordering is something that should be possible in the UI, not just by rewriting the query. Typically when I look at logs, I rapidly iterate on my view, adding columns, removing columns, rearranging them so I can see certain things side by side. This process should be zippy and friction-free.

- Breakdown: Here is an example from Kibana: https://imgur.com/Dkz6j9K. You can make the timeline display a stacked bar chart by any dimension.

- Autocomplete:

- Live view: Maybe I ran into a bug when I ran it locally; if I enabled the live tail mode and then scrolled down, then after about 2 seconds the whole page would scroll to the top even though nothing new had come in. When I try your live demo now, I see that it disables the live view when I scroll. I would argue that this is also bad UX, as it requires me to click "Resume" again. It should at least resume the mode if I scroll back up.

- Chart view: Maybe I was running into another bug that got fixed. I started latest HyperDX in Docker Compose and can't reproduce it now.

I did run into more chart bugs: Drag-to-zoom causes text to be selected, and when I zoom, the X axis range doesn't exactly match what I zoomed into, I.e. if I select from the start to the end of a curve, I expect the zoomed-in view to start and end at the same place, unless something funky is being done to the rounding of each X point. I have a video here: https://imgur.com/p6XOZ6U. [Edit: It is about granularity. I see the zoom is correct, it's just that your auto-granularity is very aggressive: https://imgur.com/C3TarFo. I've never seen this with Grafana, which bases its calculated time interval on the number of horizontally viewable data points, i.e. how many pixels wide the chart view is. Maybe an idea you should steal?]

For some reason the chart is working better for me now, but I would still argue that you're miles and miles behind Kibana. Kibana has a number of visualizations (line, bar, stacked bar, area, table, etc.; line smoothing of the kind you do is often not desirable, and a staircase is often better), and supports different aggregations such as group by top values: One thing I do constantly is to break a chart down by top K values. For example, a stacked bar chart of sum(duration) grouped by customer ID can be a very nice visualization.

I do like how HyperDX has a filter for every data series, which Kibana hides behind a convoluted UI.

- Sidebar: Yes, I suppose so!

Another thing I thought of was the filters in the sidebar. Seeing "ResourceAttributes['foo']" isn't so nice UX, when most of the fields are resource attributes, and they typically get truncated horizontally. From your demo: https://imgur.com/a/69lD3t5.

[atombender]

That's great to hear. I will take a closer look ASAP.