[marcusbuffett]

The problem I’m facing with storing roughly 600 million shorter PGNs is that the database is 100GB or so, and I’m grabbing thousands of them sort of at random. This makes the query IO bound, even though the finding the pages they’re on is virtually instant with the indexes. So a smaller database means less pages read when I do these large reads, ideally. I also have other ideas on ordering the database in a smarter way, but hoping this part helps.

[V-2]

Sure, I see your point. Obviously a wasteful format is also getting in the way of queryability. My point is that the main goal should be to improve for queryability, which inherently requires some optimizing for storage, but that's secondary. As opposed to optimizing exlusively for data size.

Because in the former case it may still be best to accept some compromise (in the form of redundancy/simplicity) to hit the sweet spot.

Especially in the context of many comments that seem to have taken an extremely "code golf"-like approach towards the problem.