Hacker News new | ask | show | jobs
by tychver 3134 days ago
Yes, but you only need one process per core, just like NodeJS.

Since 1.9 you can use real OS threads to achieve parallel IO, and certain parallel computations which can proceed without holding the Global Interpreter Lock.

JRuby offers completely unrestricted threading with a single process, plus a 3x performance boost, plus the advantage of an incredible amount of work put into their VM and GC. It's a really underrated option these days.

The problem with forking, is MRI Ruby is unaware what memory is actually inherited by forking, so eventually it causes the entire heap inherited from the parent to copy into the child.

I'm actually working on a patch to fix this. The solution is simple, just don't mark, collect, or allocate into inherited pages, but the implementation itself is fiddly.

What's really exacerbated this, is that most Linux distros now have Transparent Huge Pages turned on by default, and flipping a single bit in inherited causes a 2MB copy instead of a 4kb copy!

1 comments

I'm fairly certain this was fixed in 2.0? https://medium.com/@rcdexta/whats-the-deal-with-ruby-gc-and-...
Nah, unfortunately simply moving the GC bits from the object itself to bitmap in the page header made Ruby CoW friendlier but not CoW friendly! Each Ruby page is 4x OS pages on Linux, so marking into the header of each Ruby page still causes 1/4 of the parent heap to copy into the child process.

The bigger issue is heap fragmentation. Lots of inherited pages have a couple of free slots that Ruby will happily consume, effectively causing you to pay a 4kb copy for a 40 byte object.

This means allocating a few hundred objects can cause basically the entire heap to copy. Combine this with Transparent Huge Pages, where flipping one bit causes a 2MB copy rather than 4kb, and a few hundred object allocations can cause the entire parent process memory to be duplicated into the child.

Aaron Patterson is doing some great work to bring GC compaction to MRI, which will help reduce fragmentation, but it's a huge task.