[rnjn]

very interesting solution. and great idea to have a playground. would love to know some details on the implementation of the architecture you have shared - 1. how do you query across multiple files, do you have a query engine like data fusion doing that heavy lifting, or is this a custom implementation ? 2. how do you manage a WAL with real time query-ability across files ? have you seen any failures (recent entries missing sort of issues) Thanks, once again really interesting design and intuitively looks more economical.

[mvijaykarthik]

Thanks for your feedback, and great questions. 1. We create serverless functions to process each file and then combine the results, optimized for columnar file formats. 2. This is one of our core innovations :) We created custom representations of WALs which help us with query performance and ingesting them quickly. 3. Once a WAL is ingested, it is available for query within a few seconds. So far it has been reliable and we have not had issues with missing data.