PyPy is faster than Python, yes. But Go, C and many other (compiled) languages are way faster than PyPy. Plus, if you use a language like Go or Rust then you avoid Python's GIL and you'll have much more reasonable memory usage. Best of all, deploying is a matter of copying a binary, rather than having to deal with the absolute disaster that is Python packaging.
> Plus, if you use a language like Go or Rust then you avoid Python's GIL
No, but then you run into Go's GC and green threads. File systems fit squarely in the realms of "systems programming" (old definition [1], not new). Languages like Ada, Pascal, C/C++, Rust and D (without GC).
I wouldn't consider the workstations sold by Xerox, TI, Connecting Machines, the OS research department at ETHZ or the Microsoft’s natural language search service for the West Coast and Asia, just experiments.
With a synchronous garbage collector for cycles. Which is like the worst of both worlds, since you get the constant overhead of refcounting, plus unpredictable interruptions of unspecified duration that can happen every time a new object that might contain references to other objects is created.
To be fair, the GC can be disabled. But it's only safe to do so when you know there are no cycles, and even when such guarantee can be had for your own code, I've never seen a library guarantee that to API clients.
Maybe plenty fast for most applications but a filesystem is not one of these IMHO, especially for something as naturally resource hungry as ZFS.
A good filesystem implementation requires tight memory management and good control of what happens at the OS level. I am not saying it can't be be done in python, but it clearly isn't the right tool for the job.
I meant that for a production implementation. Python is perfectly fine for a proof of concept, in fact, it may be better than jumping straight down to C. But keeping it for production is foolish IMHO.
I was trying to speed up a log processing service running on PyPy by rewriting it in Java. I was surprised that the result was about twice slower (I know Java quite well and I didn't see obvious optimizations; most of the time was spent in GC). So it can be quite fast even in more absolute terms (VM languages), at least for some types of code.
If you manage to do it would be awesome, otherwise thanks a lot anyway for the effort :)
I’m just curious to understand why it happens because it’s exactly the opposite of what I would expect.
The only explanation that comes to my mind is excessive gc as someone else already mentioned, but it would be interesting to see the original code.
I started doing it but yes, it's too much effort to get two full benchmarks, I'm sorry :). But I think it went down to the inefficiency of String.split(): https://stackoverflow.com/questions/37007189/string-split-te... and generally the Java's String built-in methods not being GC-friendly: https://stackoverflow.com/questions/20336459/garbage-friendl.... I'm guessing that when such parts can be coded in a non-VM environment (CPython/PyPy runtime) they can be made much faster, and Java to these days has the standard library coded in pure Java?
I had a binary parser written in Python that took around 30 seconds on typical input on CPython. PyPy took that down to about 10 seconds. Rewriting it in C# took it down to 200 ms.
Not single byte, but individual fields (float32/int32/string etc). Yes, I expected a much more significant speed-up as well. It's probably because a lot of that code was driven by reflection-type techniques.
Curiously, IronPython did better than anything (but still slow). Haven't tried Jython.
Compiling the whole thing with Cython was less effective than PyPy.
I routinely fail to get speedup on PyPy. In fact I frequently get slowdowns. I imagine it's only fast if your code is slower than it needs to be to begin with.
It works well for tight loops processing much data, or heavy object-orientation (multiple levels of class hierarchies). It probably won't work well for regular Django webapps or scripts. Also, real-world Python numerical/AI code uses numpy/ML libs so there's not much to optimize in Python...
If you're going to run the file system in user space, there's no reason not to just use normal ZFS. The problem with ZFS licensing is only in combining CDDL+GPL in one unit. If you're working across the kernel/userspace boundary, there's already no problem. ZoL even already ships a fuse version that works fine.