Hacker News new | ask | show | jobs
by manigandham 3575 days ago
Why dont document databases automatically save common keys in some kind of lookup table?

Seems like a basic feature to improve space savings and processing speed.
2 comments
hendzen 3575 days ago
It's much simpler to just use an off-the-shelf compression (zlib, lz4, etc) on the database pages. This basically has the same effect, but also compresses common values.
link
malisper 3575 days ago
Postgres does use a variation of lz when using TOAST. TOAST is the technique Postgres uses to store attributes that are too big to store with the rest of the row. Unfortunately, TOAST only works within a single attribute, it doesn't work across multiple rows, or even across multiple attributes of the same row.
link
manigandham 3575 days ago
In that case, why isnt that normal and key name size a non-issue with document stores and JSON columns?

Am I missing something on why this isnt done already and automatically?

link
hendzen 3574 days ago
If you use MongoDB with the WiredTiger engine, the database pages are compressed with Snappy by default. On other systems you have the option of running the database on a filesystem that provides compression.
link
malisper 3575 days ago
My guess - it's slightly non-trivial and people are working on more important things.
link
tracker1 3575 days ago
How do you compare null, true, false, '', 'Y', 'True' in such a case, all are values that might wind up in a boolean field... though if you're using json-schema for your schemas, that's less of an issue in the common case.

IMHO anything that is to be queried against regularly should be normalized into an actual column.

link
manigandham 3575 days ago
I'm not talking about the values, just the key names themselves... databases should be smart enough to realize that "username" and "id" and "timestamp" are keys repeated in most/every record and normalize them away so there isnt as big of a storage cost.
link