My first thought was that this sounds like an idea taken from microkernels. Is this a correct analogy? Would it be sufficient to use libos and fuse to call the Linux kernel a microkernel? What would be missing?
More seriously though, you're absolutely right in that it's a step in the direction of an "optional" microkernel architecture. That's actually how rump kernels on NetBSD started: running the kernel file system driver as a library in userspace on top of a FUSE-like subsystem. It's pretty useful functionality, since it allows you to handle untrusted file system images safely in userspace, while not imposing performance penalty on the trusted images which can be handled by the same driver running in the kernel. Unlike with FUSE-specific drivers, you don't run into issues with unsymmetric driver support in userspace vs. kernel.
My first thouvht on hearing the name "libOS" was that someone had gotten Linux running on an exokernel, which is in some ways a variation on the microkernel theme.
Linux on a microkernel has been done before. The MkLinux projext is one example, though it also doubled as a port to PowerPC. I haven't heard of any exokernel attempts, though.
More seriously though, you're absolutely right in that it's a step in the direction of an "optional" microkernel architecture. That's actually how rump kernels on NetBSD started: running the kernel file system driver as a library in userspace on top of a FUSE-like subsystem. It's pretty useful functionality, since it allows you to handle untrusted file system images safely in userspace, while not imposing performance penalty on the trusted images which can be handled by the same driver running in the kernel. Unlike with FUSE-specific drivers, you don't run into issues with unsymmetric driver support in userspace vs. kernel.