[oskarpearson]

Seems like a really useful tool. Would your system support Postgres Aurora on AWS as a source database? Or does it require some lower-level access to Postgres server?

We are currently using DMS to send data to S3 and from there to Snowflake.

[saisrirampur]

PeerDB should work or Aurora PostgreSQL. It should work for both log based (CDC) and query based replication. Log based because Aurora supports pgoutput plugin. Curious, are you leveraging CDC to move data to S3? or more query (batch) based?

[oskarpearson]

We use DMS in continuous replication mode, which appears to use CDC under the hood according to https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.C...

In our setup DMS pushes Parquet files on s3. Snowflake then loads data from there.

We’ve occasionally had to do a full table sync from scratch, which is painfully slow. We are going to have to do that in the very near future - when we are upgrading from Postgres 11 to Postgres 15.

The S3 step also seems unnecessarily complicated, since we have to expire data from the bucket.

How does PeerDB handle things like schema changes? Would the change replicate to Snowflake? (I’m sure this is in the docs, but I’m supposed to be on holiday this week ) Thanks for the quick reply.

[saisrirampur]

Gotcha, that really helps. Schema changes feature is coming soon! We are actively working on it. This thread captures our thinking around it - https://news.ycombinator.com/item?id=36895220 Also have a good holiday! :)

[netvarun]

I think you were referring to this thread: https://news.ycombinator.com/item?id=36897010

[saisrirampur]

Thank you for pointing to the right thread! :)