Hacker News new | ask | show | jobs
by lunixbochs 2606 days ago
Note: this was implemented without referencing any ZFS source code and should not be subject to the CDDL.
3 comments

So we port this python to something not slow, and all the kernel-people can shut up about ZFS being terrible ;)
I can Port from Python -> Common Lisp at a rate of ~100 LoC per hour, and that's a pretty friendly port (the yield expression is the only thing that can't be just done line-by-line; truthyness is the only real "gotcha" as there are a lot of "false" values in Python, but only 1.5 in Lisp[1]).

1: I say 1.5 because there is only one false value, but it has 2 idiomatic meanings: nil (equivalent to python's None) and the empty list.

Dont forget, pypy is fast.
PyPy is faster than Python, yes. But Go, C and many other (compiled) languages are way faster than PyPy. Plus, if you use a language like Go or Rust then you avoid Python's GIL and you'll have much more reasonable memory usage. Best of all, deploying is a matter of copying a binary, rather than having to deal with the absolute disaster that is Python packaging.
> Plus, if you use a language like Go or Rust then you avoid Python's GIL

No, but then you run into Go's GC and green threads. File systems fit squarely in the realms of "systems programming" (old definition [1], not new). Languages like Ada, Pascal, C/C++, Rust and D (without GC).

[1] - https://en.wikipedia.org/wiki/System_programming_language

Filesystem with a GIL, what could possibly go wrong? /s
A lot less will go wrong than a filesystem without a GIL.

GIL is for safety and correctness, not speed.

Uh, no?

Python's global interpreter lock was added for single threaded speed and c library integrations, which often can't be used multithreaded

There was some talk about removing it recentlish to improve pythons multithreaded performance and Guido said something along the lines of

> "I'll remove it as long as single threaded performance doesn't suffer"

Which nobody succeeded in

Go? A GC'd language in kernel? (Well, yes, this has been done, from Lua to Haskell, but only experimentally.)
I wouldn't advise writing low level stuff in Go but people do enjoy a challenge from time to time: https://news.ycombinator.com/item?id=18399389
I wouldn't consider the workstations sold by Xerox, TI, Connecting Machines, the OS research department at ETHZ or the Microsoft’s natural language search service for the West Coast and Asia, just experiments.
Python is also a GC’d language...
CPython is mostly reference-counted.
>PyPy is faster than Python, yes.

Python is only slow if you use it wrong:

https://apenwarr.ca/diary/2011-10-pycodeconf-apenwarr.pdf

Maybe plenty fast for most applications but a filesystem is not one of these IMHO, especially for something as naturally resource hungry as ZFS.

A good filesystem implementation requires tight memory management and good control of what happens at the OS level. I am not saying it can't be be done in python, but it clearly isn't the right tool for the job.

I meant that for a production implementation. Python is perfectly fine for a proof of concept, in fact, it may be better than jumping straight down to C. But keeping it for production is foolish IMHO.

Faster that CPython doesn't mean it is fast.
I was trying to speed up a log processing service running on PyPy by rewriting it in Java. I was surprised that the result was about twice slower (I know Java quite well and I didn't see obvious optimizations; most of the time was spent in GC). So it can be quite fast even in more absolute terms (VM languages), at least for some types of code.
If more than 1% of your time is in GC you are doing something very wrong.
The fact that a singular implementation was better than java says less about the languages and more about the particular software.
I know that it’s asking a lot, but any chance that you can post a minimum reproducible sample? From what I know it is quite smelly...
I don't have access to this codebase now but I'll try to write some benchmark.
I had a binary parser written in Python that took around 30 seconds on typical input on CPython. PyPy took that down to about 10 seconds. Rewriting it in C# took it down to 200 ms.
If this was using a loop processing a single byte in an iteration I would expect a greater speedup on PyPy. I've seen 100x speedup in such cases.
I routinely fail to get speedup on PyPy. In fact I frequently get slowdowns. I imagine it's only fast if your code is slower than it needs to be to begin with.
It works well for tight loops processing much data, or heavy object-orientation (multiple levels of class hierarchies). It probably won't work well for regular Django webapps or scripts. Also, real-world Python numerical/AI code uses numpy/ML libs so there's not much to optimize in Python...
Only when comparing to CPython
pypy is fast, but even were this written in C, there's still the kernel-userland boundary to contend with.
Fuse
If you're going to run the file system in user space, there's no reason not to just use normal ZFS. The problem with ZFS licensing is only in combining CDDL+GPL in one unit. If you're working across the kernel/userspace boundary, there's already no problem. ZoL even already ships a fuse version that works fine.
> ZoL even already ships a fuse version that works fine.

This is not true. A FUSE implementation is wanted though: https://github.com/zfsonlinux/zfs/issues/8

Would the CDDL matter for a python implementation that will never become part of the kernel?
You'd want a reverse-engineering lawyer, so be certain. But my (IANAL) guess is: If this is a proper reverse-engineered implementation, you could then convert _this_ implementation to C, and contribute _that_ into the kernel.

Except, it seems this is BSD-licensed, so I'm not sure how that would work in the kernel (which is GPLv2).

BSDL code is fine in the GPLv2 kernel. E.g., most of the DRM drivers are dual BSD-GPL licensed.
BSD is a subset of GPL's restrictions, so you can include BSD-licensed code in a GPL work.
Subset isn't quite accurate, but "GPL compatible" might be a good way to describe it.
It's clear to me that the solution is to integrate python into the kernel.
I assume you mean this as a joke, but I would point out that at least one of the BSD family has gone and baked lua into their kernel. Granted, lua is rather meant for that kind of thing and python isn't, but it is entertaining to point out an interpreted language that has been stuck into a unix kernel:)
there was lua in linux too, with lunatik https://github.com/lunatik-ng/lunatik-ng
Bad title. It’s not a ZFS implementation, so hold your horses.
What do you mean? Can't you mount & read ZFS filesystems with this one?