[sorenjan]

I haven't done any image filtering a a good while, so I'm pretty much just speculating here. But maybe you can work on patches instead of the whole frames to save memory, since it's not a real time application anyway? I think a moving truncated mean would be better than an average of medians. Or maybe you can do one pass over the video and calculate the mean and variance for each pixel over a moving window, and then in the next pass filter out pixel values that are more than one standard deviation (or whatever you choose) from the current estimated mean and average the remaining values.

Doing linear combinations of values that aren't linear does indeed sound problematic, values from one end will have a disproportional influence over the result, although maybe it's not a big deal since the same pixel in neighboring frames will probably only vary slightly in an approximately locally linear fashion.