Hacker News new | ask | show | jobs
by itsbilal 2107 days ago
Hi, I'm on the team that works on Pebble. Partially synced WAL records are easier to detect, as they would just appear as corrupt records and we can stop WAL replay at that point. Non-WAL writes are even easier to handle as SSTable files are immutable once fully written and synced. We rely pretty heavily on fsync/fdatasync calls to guarantee that "all" the data in a given range made it.

In addition to randomized crash tests, we have a suite of end-to-end integration tests on top of Cockroach, called Roachtests, that put clusters under a combination of node crash/restart scenarios and confirm data consistency.
2 comments
ignoramous 2107 days ago
> We rely pretty heavily on fsync/fdatasync calls to guarantee that "all" the data in a given range made it.

Reminds me of fsync gate: https://news.ycombinator.com/item?id=20491965 and https://news.ycombinator.com/item?id=19119991 (not implying Pebble uses fysnc incorrectly).

link
diehunde 2106 days ago
Hi! Awesome work.

One of the cool things about the LevelDB codebase is the `Repository contents` section. A brief description of the most relevant modules so people can get familiarized with the code base quicker. As someone very interested in storage engines, I would love to see something similar here. Are you guys planning on adding some extra documentation to the project?

link