| The 32-bit ints can be solved even at the user-level library. But it's much worse than that. Even if you only need to access 2GB (or you had fixed the Java memory mapping code) you still have a .getDoucle() or .putDouble() call for every access; and that's actually a virtual call (and as far as I can tell, even though I only ever used one kind of memory channel, the JVM wouldn't inline it -- although I can't tell for sure, because the JVM also sucks at introspection). I had real computational code in C that needed to be translated to Java. First attempt (no memory mapping, converting C structs to Java objects) failed miserably because my structs were 32 bytes each, and the object overhead was 24 or 32 (don't remember), which took me beyond physical memory (using virtual memory caused a slowdown of ~1000). 2nd attempt, I switched to memory mapped arrays -- much better, only ~15 times slower. But I also had to write my own sort, because Array.sort() or whatever it was called was allocating 48 bytes for each 4 byte int to sort (wtf?), blowing memory usage up again. That's a cost people using Hadoop pay all the time -- which kind of surprises me how popular it is. You need 10 times less CPU if you do things right -- and at that scale, maintenance & hardware cost as much as salaries.... |