[pbowyer]

We're good at logging text, but how do you handle logging assets (images, audio - anything non-textual but generated) and associating them with your logs?

For example an image processing pipeline. You don't always want to log (it'd never scale) but as part of a trace you might want to keep the intermediate files so you can track down where the problem is. You've already got text logging for each step, recording metrics like duration and which filters were involved. I have saved files and referenced them in the logfile, but no log viewers I've seen understand anything beyond text. So I then have to build my own UI or open the images in turn.

Is there a pattern to handle this?

[kzh_]

+1 on existing log viewers being particularly well suited for text over non-textual assets. My experience here is limited but I believe Grafana has a dynamic image plugin if you store a link to an asset in blob storage or Base64 encode it.

I've also heard of people storing those links in a database like Snowflake then creating displays on top using Tableau or Looker, to avoid having to build a web app from scratch.

[nerdponx]

It might be interesting to have something like "statistical logging", which saves the intermediate image files 1% of the time and discards them after 30 days.

[pm90]

Maybe wild idea: generate a unique text identifier for the image + an s3 url, log that identifier rather than the image. I guess its logging metadata rather than the actual data.

[SnowHill9902]

- unused memory is wasted, you may be able to store the raw image. - if your process is deterministic store a hash. - store a low resolution image.