Hacker News new | ask | show | jobs
by loopbit 3119 days ago
> Except apparently not, because the linked article is literally saying it's not supported.

Not entirely true. Kafka, out of the box (and as far as I know, I'm no expert) will keep the records for 7 days and delete them afterwards.

Most people I know (including myself) use Kafka to keep records for longer and a good option is to use what the article describes, which is to compact the logs. In that case the log, after a configured period of time (or when it reaches determined size) gets compacted and only the latest message with an id gets saved, all previous messages with the same id get deleted (that's why the process needs a message with a null velue to perform the "delete").

Only in the case when you want to keep the data forever and can't use compaction (compaction assumes that your messages always contain the full state of an entity, so the last message will always contain the current state and the previous can be deleted with no side effects), then there's no way to delete a specific message. I'd have to read the exceptions for backups included in GDPR, but you could make the case that, in this case, the Kafka log is maintained only as a backup of the data, to be able to replay it again in case something downstream gets broken.