[elvinyung]

Here's a dumb question about log-structured systems like this: does this system work nicely with backfills? Suppose you start logging events with Waltz and you want to migrate an existing system's data into the same log. Or something goes wrong and oncall needs to manually insert old events. Does Waltz have capabilities to backfill events into the historical log or reassign transaction IDs?

This might not be needed if this is strictly used for FIFO event consumption, but I guess I was thinking of trying to make a system like this support time-sliced queries.

[refset]

These are good reasons to use a bitemporal database [1].

A log gives you "transaction time" but you need to create an efficient representation of "valid time" for backfilling and corrections.

[1] https://en.wikipedia.org/wiki/Temporal_database

Disclosure: I work on a database for Kafka that provides point-in-time bitemporal Datalog queries https://github.com/juxt/crux

[elvinyung]

Yeah, I'm currently getting bitten by various workloads that query by `created_at` instead of a canonical timestamp :) Thanks, I'll take a look!

For the "valid time" primitive I was thinking of implementing something like a hybrid logical clock that CockroachDB has (but with looser guarantees, mostly just need uniqueness and monotonicity). A sequential ID would provide a slightly nicer interface for pagination but has all the problems that I previously mentioned.