[est]

can you just buffer some writes in Vector and eliminate Kafka?

I setup some Vector to buffer ElasticSearch writes years ago, also for logs, it ran so well without any problems that I almost fogot about it.

[anticodon]

Or vice versa: make ClickHouse ingest batches directly from Kafka. Messages are already buffered in Kafka, I don't get why Vector is necessary here.

[ch2026]

tbh the only thing they needed was a correct schema that didn’t constantly spawn new parts and async_insert enabled.

[est]

https://clickhouse.com/docs/knowledgebase/kafka-to-clickhous...

For anyone if curious.