[tail_exchange]

Contextual logging makes structured logging even more powerful. For example, you can attach an ID to the contex of an http request when you receive it, which then gets logged in every operation that is performed for serving that request. If you are investigating what happened for a specific request, then you can just search for its ID.

This works with any repeatable task and identifier, like runs of a cron job, and user ids.

[ManuelKiessling]

Adding userId, clientId, sessionId, and requestId to every log line and every event in the data warehouse is one of those „one weird tricks“ that actually does improve your life.

[bigmattystyles]

Until GDPR comes for you.

[galkk]

You can track them internally (pass through in process/request flow), but have 2 version of logs: pii and non-pii, and store pii in pii logs, with much stricter access restrictions. This alone considerably mitigates problem, as often you don't need details like userid to troubleshoot.

[gregoryl]

But how does that help with compliance? You can still very easily identify the data right?

[zizee]

Isn't deleting the PII associated with a user id sufficient?

[ibotty]

Often it is not because you'll likely be able to correlate.

[llamaLord]

Using anonymous ID's that you can (manually) correlate to a real users + a reasonable log retention period is perfectly GDPR compliant and let's you retain all the benefits.

[unscaled]

Nitpick: IDs that can be correlated back to a real user is pseudonymous - not anonymous - according to GDPR[1].

You should still pseudonymize your PII wherever you can, but protection measures are required.

You should have a short term log retention period (i.e. not something like 2 years, unless you are legally required to store all the data for this period). If the log retention is too long, even if you can prove you need this data, you would still have to handle data erasure requests (a.k.a. "the right to be forgotten"). Setting up a way to delete all log entries for a certain user ID could be an alternative to a short retention period.

The other feature you want to have is encryption (both in-transit and at-rest). Encryption is not a hard requirement in GDPR, but almost no "technical" measure is. GDPR Article 32 mentions various technical measures (including encryption and pseudonymization) and requires controllers to implement them while "[t]aking into account the state of the art, the costs of implementation and the nature, scope, context and purposes of processing as well as the risk of varying likelihood and severity for the rights and freedoms of natural persons".

Here we have to do some interpretation, but generally the accepted interpretation[2] is: "state-of-the-art" refers to methods and techniques which are widely available. Cost of implementation is strongly tied to risk: high-cost technical measures are not required unless the risk is equally high.

Since encryption, pseudonymization are simple, cheap and have widely available open source implementations, it's a good idea to make sure everything is encrypted.

Restricting access control to log data is another measure that is trivial enough to implement, that it becomes a practical requirement. I've seen a lot of cases where the entire company or org had unrestricted access to logs containing PII in the past. This probably won't fly with GDPR.

Lastly, I would make sure the logging infrastructure and everything connected to it, follows industry standard security measures (at least OWASP Top 10). It sounds obvious, but I've seen a lot of cases where logs have been treated as "non-critical" part of the system ("they're just logs!") and had not been reviewed or tested for security. If you suffer from a data breach and investigation reveals your logs were not properly secured, you'll likely be fined.

So in short, if you want to play it safe, all of the below:

- Pseudonymous IDs

- Short retention period (in accordance with other laws)

- Encryption

- Restricted access

- Industry-standard security best-practices

[1] https://www.dataprotection.ie/en/dpc-guidance/anonymisation-...

[2] https://gdprhub.eu/Article_32_GDPR

[XorNot]

Except like, not really? If you need to remove someone's data for GDPR reasons, then "if match == userId then delete" is pretty straightforward in your log aggregation store.

[jcrites]

Many log aggregation stores are not optimized for performing row-level updates or deletes like this. In my experience, the majority of log aggregation stores are immutable and support primarily time-based retention only.

(Though perhaps one can meet compliance needs by keeping these logs only for a fixed maximum period of time, e.g. 30 days, and keeping only appropriately anonymized data longer.)

[wongarsu]

Saying "we need to keep these logs for 30 days to allow us to troubleshoot problems. We can't reasonably delete them sooner, but they get deleted after 30 days" is a valid way to comply. You have a justifiable reason to keep them, the interval is reasonably short, and you have good technical reasons not to do it faster.

If your internal compliance people don't like it you can also rephrase it as "we are removing the data starting right now, the procedure takes 30 days". You have one month to even respond to removal requests, and can stretch that by another two. As long as you are not intentionally causing delays these are perfectly reasonable time frames.

Of course you still have to do all the other stuff for GDPR compliance, like making sure you have rules who gets access to the log system instead of just giving it to the entire company, making sure you store to an encrypted drive, etc.

[hipadev23]

A log aggregation store that can’t handle deletes in 2024 is a product that shouldn’t be utilized. GDPR and similar redaction laws are not new.

Efficient or fast is not a requirement for GDPR, so it can happen slowly and in the background just fine.

[randerson]

A log aggregation store that can handle deletes is a security and compliance problem. Try proving to an auditor that a hacker couldn't have hacked in and then covered their tracks by deleting the logs.

[laerus]

what if you anonymize the actual user entity with that user id instead? even if you have that user id in your logs the name or any sensitive field would be something like 'GDPR says HI".

[danpalmer]

This is necessary but not sufficient. Logs can contain other data, that could be used to narrow down the user base enough that you could guess which user it is, and now from just the logs you have de-anonymised an ID and can see everything that user did, or likely did.

In reality you need multiple different steps here: anonymous IDs, well-defined reasonable retention periods, strong access control and audit logging, and a privacy policy that says why the data is collected (for service quality typically) and how/when it will be deleted.

There's no one-clever-trick to GDPR, the law was intentionally designed to require businesses to apply holistic best practice. Whether it has done that well or not is another matter, but that was at least the aim.

[llamaLord]

Exactly! The logs only need to hold ID's that you can correlate back to hydrated data later.

GDPR request comes in, just delete the record the ID refers to and you're done.

[unscaled]

This is not enough.

First, as another reply above has mentioned, other data in the logs (such as IP address, list of friends, browser fingerprint) can be used to de-anonymize the pseudonymous ID.

Second, GDPR makes it quite clear (for the reasons above) that pseudonymized data, is still considered personal data. Pseudonymization reduces the risks, but does not remove them entirely. It should generally be combined with other measures such as encryption.