Hacker News new | ask | show | jobs
by willvarfar 2300 days ago
You do not need to compress every game individually. Behind the scenes, your mainstream database is storing the data in pages, and these pages - containing many adjacent rows - can be compressed.

The compression level achieved at a page level is much higher than compressing rows individually because there is lots of repetition between rows. It also speeds up io and other good things. It is almost always good, which is why the most modern database storage engines do it and do it by default.

Consider a csv file, and compare it to the same data stored as json objects, one row per line. The uncompressed json file is going to be much bigger, as the columns are repeated in every line. But both files gzip to much the same size, because all those keys are repeated again and again and the two files have basically the same entropy.

On the other hand, compressing each line in the file individually would be a poor choice, giving relatively poor gains.

There were database engines that did row-level compression, but these performed poorly and I know of nobody who used eg innodb compression.