[stingraycharles]

Ahh I see, and that way you can quickly “remove” a whole lot of data by just removing the key, which makes for cheap operations, and/or more flexible workflow (you can periodically compact the database and remove entries for which you have no key).

Is my understanding correct?

[dalyons]

yes, but also its that a lot of the data these days ends up in pseudo-append-only stores (like s3/glacier, or many data warehouse products) where deletes/updates to old data are extremely expensive. Or just having to scan petabytes of cold stored data looking for a particular users records. Throwing away the key is instant and "free".

[chrisjc]

Interesting... this raises soooo many questions.

How are "crypto-shredding" actions propagated to the access patterns/layer?

I assume that there is an encrypted partition/cluster/shard key (in addition to similarly encrypted rows/fields) that is invalidated during the shredding causing any predicate matching on these ids to evaluate to false.

---

Now that I've typed this out, i realize that by electing to encrypt individual fields, all and any predicate matching will evaluate to false and has nothing to do with partitioning, sharding, or clustering...

I guess it would also be pretty awesome since you could invalidated entire sets of data by "shredding" grouping ids that are being used as partition/cluster/shard keys.

Now I realize that this implies that you shouldn't encrypt each and every fields of related data the same way (grouping ids), otherwise you're potentially going to end up with unique keys/ids for common attributes across sets of data... potentially rendering clustering/sharding/partition useless (cardinality too great).

While "defragging" or "rebalancing" this increasingly "sparse", old data would be expensive, surely there has to come a point where the storage costs start to exceed that of interaction costs for specific subsets of your prefixes. For instance, partitions that consist entirely of data that has had all of its respective encryption keys shredded.

---

Illuminating comment that has set my mind into overdrive... Fascinating stuff!