[sorenjan]

Try using the median or mode instead. Averaging multiple data points only works well when the data distribution is symmetrical around the true value, but the closer you get to the endpoints (black and white) the less true that assumption is. If a pixel's true value is 5, and you get some samples with values 0 to 10, and one with value 100 that outlier will not get compensated by an outlier in the opposite direction.

You could also average just the middle quantiles or similar, if you still want to average over several frames.

[an-unknown]

You are indeed right. I'm not sure how to efficiently implement a median here though, because a single frame is around 120MB large already. I assume I could/should compute the median over only a few frames to limit memory consumption and feed the result into the moving (arithmetic) average?

I'm also not entirely sure if computing the average of the non-linear "raw" sensor data which is what I did so far is a good or a bad idea / how it interacts at the endpoints of the value range.

[sorenjan]

I haven't done any image filtering a a good while, so I'm pretty much just speculating here. But maybe you can work on patches instead of the whole frames to save memory, since it's not a real time application anyway? I think a moving truncated mean would be better than an average of medians. Or maybe you can do one pass over the video and calculate the mean and variance for each pixel over a moving window, and then in the next pass filter out pixel values that are more than one standard deviation (or whatever you choose) from the current estimated mean and average the remaining values.

Doing linear combinations of values that aren't linear does indeed sound problematic, values from one end will have a disproportional influence over the result, although maybe it's not a big deal since the same pixel in neighboring frames will probably only vary slightly in an approximately locally linear fashion.