Hacker News new | ask | show | jobs
by tedunangst 1989 days ago
Does this assume the file fits in address space? That's not always the case.
4 comments

All x86-64 CPUs support 48 bit address space, with 57 bits in the newest ones [1]. It's unlikely that address space exhaustion would be a problem for the vast majority of use cases.

[1] - https://en.wikipedia.org/wiki/Intel_5-level_paging

While the default behavior of this library is to map the entire file into address space (and also the default assumption that people have regarding mmap), you can map specific portions of a file into memory to avoid virtual address exhaustion. It's just far less convenient to do so, though typically still faster than the alternatives.
According to the developers of Mongo at least, it's not worth considering. You can't store more than 2GB of data on a 32-bit machine because they map the entire datastore into memory.
From the first paragraph:

> One of the main problems a database storage engine has to solve is how to deal with data in disk that is bigger than the available memory.

I misread the comment. The file fits in address space? As sibling comment suggests, address space is v. big and it's very much a corner case (or very specific use case) for most people.