[EdSchouten]

Replying to my own comment:

I do wonder how this library deals with some of the fundamental differences beween FUSE and NFSv4. For example, with FUSE the kernel and server share intimate knowledge on which part of the file system lives in the kernel's inode cache. Only when the kernel issues a FORGET call, may the server drop information corresponding to a given nodeid.

As NFSv4 is designed to be stateless, servers may need to be able to process requests containing arbitrary file handles, regardless of how long ago they were returned as part of some prior request. You therefore see that in-kernel implementations of NFS servers rely on special file system methods to resolve objects by file handle, in addition to being able to resolve by path.

This means that if you implement FUSE on top of NFSv4, you will most likely not be able to purge any state. Your FUSE file system's FORGET method will probably never be called. This means that memory usage of fuse-t will most likely just keep on growing as time progresses? Or it announces itself as using file handle mode FH4_VOLATILE_*, but UNIX-like NFSv4 clients hardly ever know how to deal with that.

[boulos]

Nit: NFSv3 is stateless, V4 isn't (sessionid and clientid permit stateful handling of locks and shares).

I would assume this person is just doing an in-memory "NFS server" that would keep all of that state around. So it's more like a FUSE-compatible layer that speaks NFSv4 as a "front end" (since NFS clients are better than "ha ha, surprise! This FUSE backend is networked and can fail on weird ways").

I'm not sure how they chose to translate things like DELEGATE, SEQUENCE, and so on (probably just reply with an error). But for basic OPEN, READ, WRITE, etc. it's all fairly straightforward.

I had hoped they used nfs-ganesha for the NFS server / frontend, but the attributions file suggests they probably rolled their own (and thus is going to be full of bugs and immature for quite some time).

[EdSchouten]

With it being stateless, I was referring to just the part where I wanted to point out the difference: file handles are stateless, while FUSE's equivalent (nodeids) are stateful.

Regardless of whether it's an in-memory or persistent solution, the problem remains: fuse-t has little choice but to leak resources of the underlying FUSE file system, as there is no valid point in time in which you can issue FORGET operations. This means that any file system with some form of churn rate will leak memory.

Also note that delegation is effectively optional. If a server simply always replies with OPEN_DELEGATE_NONE, the client has to interact with the file as if it's stored remotely.

There is no need to implement SEQUENCE, by the way. macOS implements NFSv4.0, while SEQUENCE is part of NFSv4.1 and later.

[boulos]

> macOS implements NFSv4.0, while SEQUENCE is part of NFSv4.1 and later.

Whoa. That's a real mistake. v4.0 is widely considered a "oops, we shouldn't have released this". TIL!

[concerned_ctzn]

I don't understand the comment about leaking memory, nfs file handles don't have to be persistent. Besides, how many FUSE filesystems implement FORGET? Anyway this is the reason I went with NFSv4. I started the project with NFS v3 but then discovered many limitations: being stateless, no named attributes, questionable locking support, etc. So Eventually I dropped it and re-implemented everything on NVSv4.

[EdSchouten]

> I don't understand the comment about leaking memory, nfs file handles don't have to be persistent.

As long as the file the NFSv4 file handle refers to is still usable (i.e., linked into the file system), the NFSv4 file handle must remain usable. Note that this is not a universal requirement, but at least one that the macOS NFSv4 client enforces. It only implements FH4_PERSISTENT. This doesn't seem to be documented explicitly, but is somewhat revealed by this printf():

https://github.com/apple/darwin-xnu/blob/main/bsd/nfs/nfs_vf...

> Besides, how many FUSE filesystems implement FORGET?

Any file system that wants to remove files in the background (meaning: not by calling unlink() through the FUSE mount) must likely do proper refcounting on such files, and provide an implementation of FORGET.

For example, a file system that can give information on live football matches may want to remove files/directories belonging to matches that have already ended. In that case you want the node IDs to remain valid, but refer to files that have already been unlinked from the file system hierarchy. The FORGET operation allows you to determine when those files can be removed from the FUSE server's bookkeeping entirely.

Here is an implementation of FORGET that I wrote for Buildbarn:

https://github.com/buildbarn/bb-remote-execution/blob/master...

[concerned_ctzn]

I'm not sure I understand you point, if an inode gets removed, then GETATTR or LOOKUP would fail as it should unless you're talking about open files. In the latter case the inode will get removed when the last open handle to the file is closed

[EdSchouten]

Sure, LOOKUP would obviously fail, as the object is no longer present in the file system under any name. GETATTR is a different story. Consider this sequence of operations:

    $ mkdir foo
    $ cd foo
    $ rmdir ../foo
    $ stat .                                                                                    
    16777221 401089477 drwxr-xr-x 2 ed staff 0 64 "Sep  6 06:41:05 2022" "Sep  6 06:41:05 2022" "Sep  6 06:41:05 2022" "Sep  6 06:41:05 2022" 4096 0 0 .

Notice how we removed a directory, and were still able to obtain its attributes afterwards using GETATTR. In fact, I can even go ahead and modify some of its attributes using SETATTR:

    $ touch .
    $ stat .
    16777221 401089477 drwxr-xr-x 2 ed staff 0 64 "Sep  6 06:45:34 2022" "Sep  6 06:45:34 2022" "Sep  6 06:45:34 2022" "Sep  6 06:41:05 2022" 4096 0 0 .

So that's what FORGET is for. It allows the kernel to hold on to inodes, even if they have been unlinked from the underlying file system, regardless of whether they are opened or not.

I think it's important to realise that a FUSE nodeid _does not_ represent an identifier of an inode in the file system. Instead, they are identifiers of the inode in the kernel's inode cache. Every time the object is returned by LOOKUP, MKDIR, MKNOD, LINK, SYMLINK or CREATE, should their reference count be increased. FORGET is called to decrease it again.

[FullyFunctional]

TIL about nfs-ganesha. I have yet to digest it all, but ignoring the wonderful usability of sshfs, it seems you should be able to get the functionality of sshfs with nfs-ganesha? Am I missing something?

[fulafel]

The DB for the necessary state could also be on-disk. Maybe in practice you can expire things after a reasonably long time, like a week, even though that's not strictly spec conforming.