Hacker News new | ask | show | jobs
by tang8330 1053 days ago
Thanks for the comment!

Your comment regarding DDL is interesting.

Today, this is what happens:

1/ Column doesn't exist in the destination, we'll create it based on our typing inference from the data type (important: not the data value).

2/ Certain tools will handle automatic column data type conversion if a change like this was detected at the source. We do not do this. We will simply hard fail and cause head-of-line blocking reasons being: this is anti-pattern and should be rare, in which case - it's okay to cause an err and require manual intervention for this breaking change.

3/ If the column has been dropped from the source, you as the end user can decide whether this column should be also dropped in the destination, or not. The default is not to drop it.

^ We hear more customers explicitly don't want columns to be dropped because it could cause downstream errors, such as other views / tables not compiling due to referencing a non-existent column.

We haven't heard much from folks that don't even want columns to be added. If there is a need, we can definitely add that as a config option to provide maximum configurability.

> Finally, the biggest issue with CDC always ends up being the seed loads, recoveries and the incremental snapshot strategies.

Yep totally. On the recovery bit, this is exactly why we are leveraging Kafka. If there are any particular issues, we simply don't commit the offset and cause head-of-line blocking.

On the incremental snapshot and recoveries bit, we primarily leverage Debezium's DDD-3 high watermark strategy [1] for MySQL and MongoDB. Postgres has a different issue in that replication slots can grow really fast, esp on AWS! [2]. We ended up writing our own custom snapshotter for Postgres that is Debezium compatible to onboard customers that have a massive dataset and cannot afford to have a read lock on their WAL.

[1] https://github.com/debezium/debezium-design-documents/blob/m... [2] https://www.morling.dev/blog/insatiable-postgres-replication...

2 comments

The custom snapshotter sounds interesting, potentially a good selling point. On recovery end, in my designs I have also found it useful to have synthetic events so in breakage of CDC I can stitch logs together and not just start from scratch and lose history. I can see you are in the depths of it, more than I’ve been for a while. Wish you the best.
> Postgres has a different issue in that replication slots can grow really fast, esp on AWS! [2]. We ended up writing our own custom snapshotter for Postgres that is Debezium compatible to onboard customers that have a massive dataset and cannot afford to have a read lock on their WAL.

Does Debezium's DDD-3 watermark (DBLog) implementation for Postgres not process the WAL quickly enough? We don't use it ourselves either, but architecturally it appears it would reasonably bound how long the WAL can remain un-read?

Agreed that many production DBs people care about have pretty severe limitations here! Managed Supabase is another good example.

On a single unbounded (CPU + mem) Debezium running on a VM extracting Postgres, I was able to clock in about 7-10m/hr. You could increase the # of tasks, but then it'll hinder your DB perf. Also, this is on your primary DB.

We found it far more efficient and less risky to do CDC streaming and snapshotting w/o read lock in parallel to two different topics. Once snapshot is done and drained, we then move to drain the CDC topic.