[pjtr]

Since the order of the samples doesn't matter, could you sort them somehow so the Gzipped stream of samples can be compressed better? (E.g. sort the color index by component average, by red, by maximum component, ... and sort the point index by color.)

Have you tried struct-of-array (AAA...BBB...) instead of array-of-struct (ABABAB...) layouts?

[ruarai]

I tried your struct-of-array idea, and that's produced an okay improvement ~1%.

Sorting them seems tough as the index of each value must match for each channel, so any sorting would have to occur beforehand. Except that is already sorted by x-y values, and my attempts otherwise have failed to produce results.

[pjtr]

I'm missing where they are sorted by x-y values.

I did a quick and dirty experiment: http://pastebin.com/NjZNRjw1

Seems about 30% smaller than before on http://i.imgur.com/5zwCEF5.png

[ruarai]

Wow, that works pretty well. I was mistaken in thinking that either the Dictionary class or the process of sampling would sort them.

Mind if I merge that? Or you could submit a pull request. Either would be great!

Also, do you know of any resources for learning about how to optimise for gzip compression? Google is just telling me about compression for websites.

[pjtr]

Sure, feel free to merge.

I don't know any resources specifically about gzip compression. Demosceners have very practical and fun compression know how, so maybe look into: http://www.farbrausch.com/~fg/seminars/workcompression.html

[willvarfar]

Compression gurus hang out at http://encode.ru

You'll likely find a compressor much more suited to your particular data than gzip.