[twoodfin]

Like danso, I admire the detective-work here. I would like to point out, though, that XCode's Instruments utility has a fantastically useful "Leaks" mode that will identify leaked allocations, including a stack trace. It can attach to a running process and has a non-disastrous impact on performance, though like most such tools it's voracious for memory.

Other platforms likely have similar tools, though I have yet to stumble across one as easy to use.

[nelhage]

(author here) Yeah, there are a lot of such tools, and they can be invaluable. In my experience, though, if you're working with a large, complex system, including pieces like Ruby and OpenSSL which do some grody tricks internally, they'll flag lots of false positives and noise, and it takes a lot of work to get a development environment where they work well.

Also, we weren't yet sure whether this bug was reproducible in development/QA, or was only triggerable in production, so using techniques that worked directly on the running image was attractive.

[FuzzyDunlop]

I actually think there's something more remarkable about figuring this out by hand, in the command line. And writing about it, just the fact that you can capture the exact steps taken with a simple copy and paste blows the GUI approach well out of the water.

This was the work of someone who knew what they were doing, of course. Had it been, say, me, then the effect and utility of the piece would have been considerably lacking.

[kevingadd]

On Windows, there's kernel-mode support for similar allocation instrumentation, where every heap allocation can be tagged with a stacktrace.

I wrote a frontend: https://github.com/kevingadd/HeapProfiler but you can also just invoke it manually from the shell and then capture the state of the heap at your leisure.