[bob1029]

> The problem is always growth, either GC jank from a massive heap, running out of RAM, or those loops eventually catching up with you

Absolutely. The challenge is having enough faith that it will take long enough to catch up to you.

Statistically speaking, it won't catch up to you and if it does, it will take so long you should have seen it coming from miles away and had time to prepare.

In my systems that use an in-memory/append-only technique, I try to keep only the pointers and basic indexes in memory. With modern PCIe flash storage, there is no good justification for keeping big fat blobs around in memory anymore.

[aprdm]

Could you expand what you mean by keeping pointers and basic index in memory?

[bob1029]

Pointers are tuples of (Id, LogOffset) and are used to map logical identities to positions of those objects in the append-only log.

Indexes are usually a tuple of (Some64BitKey, Id) and are used to map physical business keys to logical object identities. These entries are only candidates in the case where the key material needs to be hashed and inspected for actual equivalence.

One big advantage with this approach is that you can stream big blobs directly out of the log to a caller-supplied buffer or stream. No intermediate allocations required aside from some small buffers.

[aprdm]

Awesome, thank you for sharing!