Hacker News new | ask | show | jobs
by hongalex 2093 days ago
By the way, in addition to ordered messages, Pub/Sub has recently gotten a number of new features including message filtering, dead letter queues, and retry policy (some GA, some beta). You can find out more here: https://cloud.google.com/pubsub/docs/release-notes

Disclaimer: I work on Pub/Sub
2 comments
dbb01 2093 days ago
Having these features built in without having to hack them together ourselves is nothing short of wonderful. Thanks for the great additions!
link
andrewmutz 2092 days ago
Any plans for permanent message retention?

Kafka can be configured to store messages forever. This means it can be used as the canonical store of both current and past data. It is attractive for some applications.

Any plans to add the ability to retain messages forever? I know I can store them in GCS and replay, but that's not what I'm looking for.

link
pradn 2092 days ago
I can see how this would be useful. You could use the snapshot/seek features to replay messages from a particular point. It'd be useful retain messages for a long time so you could replay messages from any point. I could even imagine a feature for replaying back messages from only a time range. We're also aware of requests for more seamless integration with other GCP services; they may augment or provide a similar set of features. I can't comment on timelines unfortunately. :)

Disclaimer: I work on Cloud Pub/Sub, but this is my own opinion.

link
andrewmutz 2092 days ago
That makes sense. To add some color to the use case:

We'd like to use a pub sub system to store binlogs from databases (as done in projects like https://debezium.io/).

We'd like any team to be able to bring a replica database online by starting from the beginning of time and playing back the binlogs. And then to keep playing any future binlog messages to keep the replica current.

For usage like this, we could in theory keep all past messages on GCS, and then access the old messages on GCS and then when those are up to date, get the messages from pubsub. Or we could keep them on GCS and replay them on pubsub in the future if we wanted to.

But in a perfect world, I'd prefer the pubsub system to handle this, and allow a consumer to start at the beginning of time and get up to date, and stay up to date in a simple way.

link
pradn 2092 days ago
Thanks for your explanation. It makes a lot of sense. :)
link