Hacker News new | ask | show | jobs
by jsty 1101 days ago
In the watermarks documentation it mentions that events arriving after the watermark are dropped. Are there any plans to make this configurable (to disable dropping or trigger exception handling) and/or alertable?

I can think of quite a few use cases (particularly in finance) where we'd want late-arrivals to be recorded and possibly incorporated into later or revised results, not silently dropped on the floor.

1 comments

Yeah, currently late-arriving data is dropped but we will be making this more configurable. We're currently working on what we call "update tables," which means being able to emit incremental changes to state as well as final results. Once that's in we'll be able to give richer semantics around late arriving data.