[avinassh]

I don't get this scheme at all. The protocol violates durability, because once the client receives success from server, it should be durable. However, completion record is async, it is possible that it never completes and server crashes.

During recovery, since the server applies only the operations which have both records, you will not recover a record which was successful to the client.

[benjiro]

I think you missed the part in the middle:

-----------------

So the protocol ends up becoming:

Write intent record (async) Perform operation in memory Write completion record (async) Return success to client

-----------------

In other words, the client only knows its a success when both wal files have been written.

The goal is not to provide faster responses to the client, on the first intent record, but to ensure that the system is not stuck with I/O Waiting on fsync requests.

When you write a ton of data to database, you often see that its not the core writes but the I/O > fsync that eat a ton of your resources. Cutting back on that mess, results that you can push more performance out of a write heavy server.

[loeg]

No, we saw this scheme, it just doesn't work. Either of the async writes can fail after ack'ing the logical write to the client as successful (e.g., kernel crash or power failure) and then you have lost data.

[cyanydeez]

You can always have data loss. The intent is that when the client is told the data is saved, it doesnt happen before the garuntee.

I dont know if OP achieved this, but the client isnt told "we have your data" until both of the WALs are agreeing. If the system goes down those WALs are used to rebuild data in flight.

The speed up allows for decoupling synchronous disk writes that are now parallel.

You are not conceptualizing what data loss means in the ACID contract between DB and Client.

But you

[loeg]

> I dont know if OP achieved this,

They did not.

> but the client isnt told "we have your data" until both of the WALs are agreeing.

Wrong. In the proposed scheme, the client writes are ack'd before the WAL writes are flushed. Their contents may or may not agree after subsequent power loss or kernel crash.

(It is generally considered unacceptable for network databases/filers to be lossier than the underlying media. Sometimes stronger guarantees are required/provided, but that is usually the minimum.)

[jcgrillo]

There's no fsync in the async version, though, unless I missed it? The problem with the two WAL approach is that now none of the WAL writes are durable--you could encounter a situation where a client reads an entry on the completion WAL which upon recovery does not exist on disk. Before with the single fsynced WAL, writes were durably persisted.