[convolvatron]

if there is a hole in the log then the end of the log is before the hole. you do have to have checksums on log chunks, and better a kind of rolling hash, but you're really just talking about he number of entires that we would have liked to commit but didn't

[mightyham]

Yeah this is a good point, and maybe a hole wasn't the right way to explain myself. The point is that the way a WAL is supposed to work is that the main data store always lags behind the WAL, so that if a partial operation (always idempotent) occurs on shutdown it is replayed on start up and fixed. In the case I describe, because of a lack of fsync it's possible for the WAL to lag the main data store, so partial operations will not be fixed on start up.

[convolvatron]

that's a much more interesting problem. fundamentally we're in a bad position by having two different formats, one optimized for writing and one for reading, that admit inconsistency between them. Postgres mitigates this slightly by having page level updates to the read indices also be present in the log (physiological), but that's always seemed like a huge waste to me.

if we give ourselves two definitions of persisted - logically(wal or write) and physically (index or read), it seems like we can maintain the invariant that P < L. (1) by keeping an in memory view of P-L that we have to consult on every read to assert eh delta and (2) an expensive but asynchronous flush path for updating P driven from reads verifying L has landed, then have we patched all the holes(?).

edit: of course one of the root problems here is the drive lying, so how can we understand that some log block has actually commit so that we can update P