Hacker News new | ask | show | jobs
by anamax 1 day ago
> You can't get the inability to allocate some large object because there's a spray of small objects in its way ... The large objects live in their own space

Assume for the sake of argument that the large object space is for objects >=1MB.

Allocate lots of 1MB objects then free every other one (by address).

Unless you're willing to let the large object space grow without bounds....

3 comments

There is a theorem that is often stated in operating system courses as "For any possible allocation algorithm, there exist streams of allocation and deallocation requests that defeat the allocator and force it into severe fragmentation.". You can ask your friendly local AI about "Bounds for Some Functions Concerning Dynamic Storage Allocation" by J. M. Robson in the July 1974 Journal of the ACM and some various follow-up papers. For any non-compacting memory management algorithm, including traditional malloc/free, there will always be a way of defeating it and forcing it to fragment.

However, consider that Go has been in production for 14 years now, and one of its bread-and-butter applications is network servers, which will collectively exercise quite a bit of the memory allocation pattern space, including some pathological aspects of it. You should expect to need to do better than that to really throw it for a loop.

While the theorem proves some such sequence exists, there's no guarantee that the sequences will be easy to describe in some sort of English sentence.

Large objects (≥ 32 KiB) does not use these categorized arenas, they instead allocate an integer number of 8KiB pages from a heap.
I haven’t read the article, but for allocations that large, chances are they get allocated as entire memory pages and the garbage collector returns that memory to the OS.

Also, even if it doesn’t, in a 64-bit address space it takes lots of 1MB objects to make that cause problems (there’s room for over 10¹⁶ of such objects)

Correction: over 10¹³. Still a lot.
Sounds like a challenge.