[EdSchouten]

What a coincidence! For a project that I maintain (Buildbarn, a distributed build cluster for Bazel) I recently generalized all of the FUSE code I had into a generic VFS that can both be exposed over FUSE and NFSv4. The intent was the same: to provide a better out of the box experience on macOS. Here's a design doc I wrote on that change. Slight warning that it's written with some Buildbarn knowledge in mind.

https://github.com/buildbarn/bb-adrs/blob/master/0009-nfsv4....

Fortunately, fuse-t doesn't make any of my work unnecessary. Buildbarn uses go-fuse, which talks to the FUSE character directly instead of using libfuse. fuse-t would thus not be a drop-in replacement. Phew!

PS: A bit unfortunate that fuse-t isn't Open Source. :-(

[EdSchouten]

Replying to my own comment:

I do wonder how this library deals with some of the fundamental differences beween FUSE and NFSv4. For example, with FUSE the kernel and server share intimate knowledge on which part of the file system lives in the kernel's inode cache. Only when the kernel issues a FORGET call, may the server drop information corresponding to a given nodeid.

As NFSv4 is designed to be stateless, servers may need to be able to process requests containing arbitrary file handles, regardless of how long ago they were returned as part of some prior request. You therefore see that in-kernel implementations of NFS servers rely on special file system methods to resolve objects by file handle, in addition to being able to resolve by path.

This means that if you implement FUSE on top of NFSv4, you will most likely not be able to purge any state. Your FUSE file system's FORGET method will probably never be called. This means that memory usage of fuse-t will most likely just keep on growing as time progresses? Or it announces itself as using file handle mode FH4_VOLATILE_*, but UNIX-like NFSv4 clients hardly ever know how to deal with that.

[boulos]

Nit: NFSv3 is stateless, V4 isn't (sessionid and clientid permit stateful handling of locks and shares).

I would assume this person is just doing an in-memory "NFS server" that would keep all of that state around. So it's more like a FUSE-compatible layer that speaks NFSv4 as a "front end" (since NFS clients are better than "ha ha, surprise! This FUSE backend is networked and can fail on weird ways").

I'm not sure how they chose to translate things like DELEGATE, SEQUENCE, and so on (probably just reply with an error). But for basic OPEN, READ, WRITE, etc. it's all fairly straightforward.

I had hoped they used nfs-ganesha for the NFS server / frontend, but the attributions file suggests they probably rolled their own (and thus is going to be full of bugs and immature for quite some time).

[EdSchouten]

With it being stateless, I was referring to just the part where I wanted to point out the difference: file handles are stateless, while FUSE's equivalent (nodeids) are stateful.

Regardless of whether it's an in-memory or persistent solution, the problem remains: fuse-t has little choice but to leak resources of the underlying FUSE file system, as there is no valid point in time in which you can issue FORGET operations. This means that any file system with some form of churn rate will leak memory.

Also note that delegation is effectively optional. If a server simply always replies with OPEN_DELEGATE_NONE, the client has to interact with the file as if it's stored remotely.

There is no need to implement SEQUENCE, by the way. macOS implements NFSv4.0, while SEQUENCE is part of NFSv4.1 and later.

[boulos]

> macOS implements NFSv4.0, while SEQUENCE is part of NFSv4.1 and later.

Whoa. That's a real mistake. v4.0 is widely considered a "oops, we shouldn't have released this". TIL!

[concerned_ctzn]

I don't understand the comment about leaking memory, nfs file handles don't have to be persistent. Besides, how many FUSE filesystems implement FORGET? Anyway this is the reason I went with NFSv4. I started the project with NFS v3 but then discovered many limitations: being stateless, no named attributes, questionable locking support, etc. So Eventually I dropped it and re-implemented everything on NVSv4.

[EdSchouten]

> I don't understand the comment about leaking memory, nfs file handles don't have to be persistent.

As long as the file the NFSv4 file handle refers to is still usable (i.e., linked into the file system), the NFSv4 file handle must remain usable. Note that this is not a universal requirement, but at least one that the macOS NFSv4 client enforces. It only implements FH4_PERSISTENT. This doesn't seem to be documented explicitly, but is somewhat revealed by this printf():

https://github.com/apple/darwin-xnu/blob/main/bsd/nfs/nfs_vf...

> Besides, how many FUSE filesystems implement FORGET?

Any file system that wants to remove files in the background (meaning: not by calling unlink() through the FUSE mount) must likely do proper refcounting on such files, and provide an implementation of FORGET.

For example, a file system that can give information on live football matches may want to remove files/directories belonging to matches that have already ended. In that case you want the node IDs to remain valid, but refer to files that have already been unlinked from the file system hierarchy. The FORGET operation allows you to determine when those files can be removed from the FUSE server's bookkeeping entirely.

Here is an implementation of FORGET that I wrote for Buildbarn:

https://github.com/buildbarn/bb-remote-execution/blob/master...

[concerned_ctzn]

I'm not sure I understand you point, if an inode gets removed, then GETATTR or LOOKUP would fail as it should unless you're talking about open files. In the latter case the inode will get removed when the last open handle to the file is closed

[FullyFunctional]

TIL about nfs-ganesha. I have yet to digest it all, but ignoring the wonderful usability of sshfs, it seems you should be able to get the functionality of sshfs with nfs-ganesha? Am I missing something?

[fulafel]

The DB for the necessary state could also be on-disk. Maybe in practice you can expire things after a reasonably long time, like a week, even though that's not strictly spec conforming.

[justsomeadvice0]

You might look at webdav instead, which has actual existing library implementations (unlike NFSv4), supports user/pass auth (in some cases you don't want wide-open services on loopback) and is based on HTTP; that would make the whole thing easier. Various projects offer a webdav combat layer, see cryptomator for one example: https://docs.cryptomator.org/en/latest/desktop/vault-mountin...

I went down the NFS road long ago - I somewhat (??) remember that coaxing macOS to mount_nfs on loopback is actually a PITA (I remember having to activate 127.0.0.* and mount 127.0.0.2).

[skissane]

I remember, around 10 years ago, mucking around with WebDAV, and discovering many WebDAV clients were extremely crapulent. Especially on Windows.

Maybe, some years later, Windows' WebDAV client has improved. And I can't remember whether macOS WebDAV was better or not.

[EdSchouten]

For Buildbarn I need to provide a writable file system that acts in a POSIX compliant way. I'm not convinced WebDAV could provide that.

[justsomeadvice0]

Fair point, agreed I doubt mount_webdav ends up posix compliant.