[andrewmutz]

Any plans for permanent message retention?

Kafka can be configured to store messages forever. This means it can be used as the canonical store of both current and past data. It is attractive for some applications.

Any plans to add the ability to retain messages forever? I know I can store them in GCS and replay, but that's not what I'm looking for.

[pradn]

I can see how this would be useful. You could use the snapshot/seek features to replay messages from a particular point. It'd be useful retain messages for a long time so you could replay messages from any point. I could even imagine a feature for replaying back messages from only a time range. We're also aware of requests for more seamless integration with other GCP services; they may augment or provide a similar set of features. I can't comment on timelines unfortunately. :)

Disclaimer: I work on Cloud Pub/Sub, but this is my own opinion.

[andrewmutz]

That makes sense. To add some color to the use case:

We'd like to use a pub sub system to store binlogs from databases (as done in projects like https://debezium.io/).

We'd like any team to be able to bring a replica database online by starting from the beginning of time and playing back the binlogs. And then to keep playing any future binlog messages to keep the replica current.

For usage like this, we could in theory keep all past messages on GCS, and then access the old messages on GCS and then when those are up to date, get the messages from pubsub. Or we could keep them on GCS and replay them on pubsub in the future if we wanted to.

But in a perfect world, I'd prefer the pubsub system to handle this, and allow a consumer to start at the beginning of time and get up to date, and stay up to date in a simple way.

[pradn]

Thanks for your explanation. It makes a lot of sense. :)