[jcrites]

Many log aggregation stores are not optimized for performing row-level updates or deletes like this. In my experience, the majority of log aggregation stores are immutable and support primarily time-based retention only.

(Though perhaps one can meet compliance needs by keeping these logs only for a fixed maximum period of time, e.g. 30 days, and keeping only appropriately anonymized data longer.)

[wongarsu]

Saying "we need to keep these logs for 30 days to allow us to troubleshoot problems. We can't reasonably delete them sooner, but they get deleted after 30 days" is a valid way to comply. You have a justifiable reason to keep them, the interval is reasonably short, and you have good technical reasons not to do it faster.

If your internal compliance people don't like it you can also rephrase it as "we are removing the data starting right now, the procedure takes 30 days". You have one month to even respond to removal requests, and can stretch that by another two. As long as you are not intentionally causing delays these are perfectly reasonable time frames.

Of course you still have to do all the other stuff for GDPR compliance, like making sure you have rules who gets access to the log system instead of just giving it to the entire company, making sure you store to an encrypted drive, etc.

[hipadev23]

A log aggregation store that can’t handle deletes in 2024 is a product that shouldn’t be utilized. GDPR and similar redaction laws are not new.

Efficient or fast is not a requirement for GDPR, so it can happen slowly and in the background just fine.

[randerson]

A log aggregation store that can handle deletes is a security and compliance problem. Try proving to an auditor that a hacker couldn't have hacked in and then covered their tracks by deleting the logs.

[hipadev23]

That’s an incredibly weak response. Laws you can’t fuck with, auditors can fuck off. I’d love you trying to explain to the EU why you’re violating their laws because some auditor wanted to check a box. I sure hope your auditors are assuming legal responsibility.

[randerson]

Don't log anything you're not allowed to log. But in some industries (like finance) you need an immutable logging system and if you could easily delete evidence of a crime or security breach that would be a bug not a feature.

[thanksgiving]

I don’t understand this… what if we had no logs?

[randerson]

I should have mentioned this is really only an issue if your business has regulatory requirements that necessitate tamper-proof logging.