[Nebyou]

This example is a full refresh, but incremental is usually the norm, especially for our supported connectors and continuous syncs. Our models can detect incremental columns.

Audit timestamps (usually tables) are typically created in intermediary stages (whose materializations you would have access to in your database) before getting pruned out to fit your destination schema. Of course, if the destination schema expects these audit tables or columns, they would be included in the target.

To your last question, if you include these tables or columns in the end schema you specify to Lume (or create a separate flow with a new end schema with these fields), what you described is definitely possible.

[gregw2]

How does Lume know what to fill in on the target table if a record effective end date isn’t in the source system but is a property of when the data was fetched? How does Lume know to update the record effective termination date of a target row when a new row comes along?

(This type of data modeling is described here: https://www.kimballgroup.com/data-warehouse-business-intelli... )

[Nebyou]

Apologies, I misread your earlier question. Lume can only ascertain information from what is given directly or from what assumptions it can make with reasonable confidence. So, in this case, this will not be possible unless there is any information in the start schema where this can be interpolated. If these prior transformations were done in a dbt project, we could extract the information needed for this easily (a dbt project can be integrated and represented as a separate flow within your workspace).