Does ZFS or any other FS offer special operations which DB engine like RocksDB, SQLite or PostgreSQL could benefit from if they decided to target that FS specifically?
Internally, ZFS is kinda like an object store[1], and there was a project trying to expose the ZFS internals through an object store API rather than through a filesystem API.
Sadly I can't seem to find the presentation or recall the name of the project.
On the other hand, looking at for example RocksDB[2]:
File system operations are not atomic, and are susceptible to inconsistencies in the event of system failure. Even with journaling turned on, file systems do not guarantee consistency on unclean restart. POSIX file system does not support atomic batching of operations either. Hence, it is not possible to rely on metadata embedded in RocksDB datastore files to reconstruct the last consistent state of the RocksDB on restart. RocksDB has a built-in mechanism to overcome these limitations of POSIX file system [...]
ZFS does provide atomic operations internally[1], so if exposed it seems something like RocksDB could take advantage of that and forego all the complexity mentioned above.
How much that would help I don't know though, but seems potentially interesting at first glance.
Sadly I can't seem to find the presentation or recall the name of the project.
On the other hand, looking at for example RocksDB[2]:
File system operations are not atomic, and are susceptible to inconsistencies in the event of system failure. Even with journaling turned on, file systems do not guarantee consistency on unclean restart. POSIX file system does not support atomic batching of operations either. Hence, it is not possible to rely on metadata embedded in RocksDB datastore files to reconstruct the last consistent state of the RocksDB on restart. RocksDB has a built-in mechanism to overcome these limitations of POSIX file system [...]
ZFS does provide atomic operations internally[1], so if exposed it seems something like RocksDB could take advantage of that and forego all the complexity mentioned above.
How much that would help I don't know though, but seems potentially interesting at first glance.
[1]: https://youtu.be/MsY-BafQgj4?t=442
[2]: https://github.com/facebook/rocksdb/wiki/MANIFEST