[PhantomGremlin]

He mentioned "several" issues. One was Intel Ethernet, but the other was a serious issue with the kernel:

on a server with 128 GB of RAM, over 70 GB of RAM was being held down by the kernel and left idle for an extended time. As you can imagine, this didn't help the ZFS ARC size, which got choked down to 20 GB or so.

That's a big issue. Is he supposed to periodically reboot his NFS servers to free up the idle RAM?

[bcantrill]

Here too we have data that borders on the anecdotal: he comes to conclusions first and then seems to look for data to backfill them. Should, as Garrett suggested, we adopt high water marks on magazines? Perhaps. Would this issue have addressed his situation? Unknown, because (once again) we have an issue that isn't being seen widely, represents suboptimal behavior (in contrast to fatal behavior), and lacks the hard data to be able to allow it to be definitively root-caused. Not a recipe for success, in any community.

[shiftoutbox]

I could be wrong but isn't this the normal memory management in Solaris an its derivatives? Even if it's not arc and it was used and now marked as inactive . The kernel keep its in a different "accounting bucket" , but it's still usable . What I suspect the op is ticked off with is actually arc back pressure and l1 arc eviction which can be horridly slow .