[Dylan16807]

You ask why it should work differently but then give a good reason why it should work differently: sometimes splitting by bytes is not the best unit.

And it actually does often work differently for PNG! PNG has a handful of preprocessing options for the pixels. So in filter mode 2, for example, deflate is encoding the difference between each pixel and the pixel above it. More or less.

[twic]

But you still apply Huffman coding to the bytes after filtering.

I think this is true even when using 16 bits per sample, but i'm not sure.

From what i remember, in gzip, Huffman coding is applied to the offsets of repeated blocks, which are not single bytes.