[gavinray]

Please don't do this.

Have the OS handle memory paging and buffering for you and then use Java's parallel algorithms to do concurrent processing.

Create a "MappedByteBuffer" and mmap the file into memory.

If the file is too large, use an "AsynchronousFileChannel" and asynchronously read + process segments of the buffer.

[papercrane]

If you're using a newer JVM you can also map a "MemorySegment", which doesn't have the 2GiB limit that byte buffers have.

[gavinray]

Good point, have written about this in the past

https://gavinray97.github.io/blog/panama-not-so-foreign-memo...

[switchbak]

Memory mapping is fun, but shouldn't we have some kind of async IO / uring support by now? If you're looking at really high-perf I/O, mmaping isn't really state of the art right now.

Then again, if you're in Java/JVM land you're probably not building bleeding edge DBs ala ScyllaDB. But I'm somewhat surprised at the lack of projects in this space. One would think this would pair well with some of the reactive stream implementations so that you wouldn't have to reimplement things like backpressure, etc.

[threeseed]

a) There have been libraries supporting io_uring on the JVM for many years now.

b) SycllaDB is not bleeding edge. It uses the relatively old now DPDK.

c) There are countless reactive stream implementations e.g. https://vertx.io/docs/vertx-reactive-streams/java/

[switchbak]

Compared to what the JVM offers, Syclla is certainly way ahead - happy to hear what the latest greatest approaches are.

I'm very aware of various reactive stream impls - I was saying that this work should plug into them rather than reinventing the wheel.

[hawk_]

I thought DPDK would still be faster than io_uring.

[jlokier]

Last time I measured on Linux (a few years ago), with NVMe, mmap + calling out to a thread pool to async-page-touch (so the main thread didn't block) was faster than io_uring (from the main thread) for random access reads.

[SillyUsername]

Better caveat that with, "but watch memory consumption, given the nature of the likes of CopyOnWriteArraylist". GC will be a bitch.

[mprataps]

Thanks for this comment. This will be an interesting aspect to explore.