[siddharthgoel88]

For Java applications, we built a structured logging library which would do a few things -

  - Add OTel based instrumentation to generate traces
  - Do salted hash of PII (injected in plain text by API Gateway in each request) like userid, etc to propagate internally to other downstream services via Baggage
  - Inject all this context like trace-id and hashed PIIs into log
  - Have Log4j and Logback Layout implementations to structure logs in JSON format

Logs are compressed and ingested to AWS S3 so it is also not expensive to store so much logs to S3.

AWS provides a tool called S3Select to search structured logs/info in S3. We built a Golang Cobra based cli tool, which is aware of the structure we have defined and allows us to search for logs in all possible ways, even with PII info even without saving.

In just 2 months, with 2 people we were able to build this stack and integrate to 100+ microservices and get rid of Cloudwatch. This not just saved us a lots of money on Cloudwatch side but also improved our capability to search to logs with a lot of context when issues happens.

[e3bc54b2]

hey, we're in pretty similar place logging wise, and I would really like to know more about your solution. If at all possible, I'd like to understand your rationale and implementation architecture more.

[siddharthgoel88]

Next month I will be publishing a blog on this topic. I will share the link here as well.