Hacker News new | ask | show | jobs
by juancampa 583 days ago
We're also interested in SQLite shared by multiple processes on something like Regatta but my concerns are the issues described in the SQLite documentation about NFS [1]. Notably "SQLite relies on exclusive locks for write operations, and those have been known to operate incorrectly for some network filesystems."

[1] https://sqlite.org/useovernet.html
1 comments
huntaub 583 days ago
Ah, yes — there are some specific file locking concerns with NFSv3 (notably that locks aren’t built as leases like in NFSv4). Let me do a double click here, but I know we will be able to support locks correctly with our custom protocol when we launch it by the end of the year.
link
juancampa 583 days ago
One more question. How does it handle large files that are frequently modified in arbitrary locations (like a SQLite file)? Will it only upload the "diffs" to S3? I'm guessing it doesn't have to scan the whole file to determine what's changed since it can keep track of what's "dirty".

I ask because last time I checked, S3 wouldn't let you "patch" an object. So you'd have to push the diff as separate objects and then "reconstruct" the original file client-side as different chunks are read, right?

link
huntaub 583 days ago
That's correct re: the S3 API. What we do is we "merge" multiple write requests together to minimize the cost to you and the number of requests to S3. For example, if you write a file 1,000 times in the span of a minute, we would merge that into a single PutObject request to S3. Of course, we force flush the data every few minutes (even if it's being written frequently) in order to make sure that there's an up-to-date copy in S3.
link
mdaniel 583 days ago
I would really enjoy hearing why SMBv4 or the hundreds of other protocols are somehow insufficient for your needs. The thought of "how hard can a custom protocol be?!" makes me shudder, to say nothing of the burden -- ours and yours -- of maintaining endpoint implementations for all the bazillions of places one would want to consume a network mount
link
huntaub 583 days ago
Ultimately, we're just working on a different problem space than these protocols. That's not to say that all of the existing protocols are bad, I absolutely believe that these protocols are great. Our ultimate goal, though, is to replace block storage, with a file-layer protocol. This sort of requires different semantics than what the existing file protocols support.

I don't at all disagree that it's a hard problem! That's part of what makes it so fun to work on.

link