[danso]

There are two things I really like about this walkthrough:

1. It shows how bugs can be something quite conceptually simple

2. It shows the value of logical, detective-like thinking in tracking these bugs.

Even as a programmer, I still think of bug-hunting as something requiring an encyclopedia knowledge of the trivial and arcane. Obviously, it looks easier in hindsight, but the OP does a great job of demonstrating how you can discover a much-overlooked flaw with the right deductive thinking (and experience with profiling tools)

[icambron]

Deduction is great, but what I read here was how much experience matters in debugging. You have to know what to look and test for to provide fodder for your reasoning capabilities, or else you're just shooting in the dark. The OP here had a plan informed by what I'm guessing are years of experience solving this kind of thing ("It's probably small objects. I should dump the core and look for repeating memory patterns."). Then there's a lot of accumulated wisdom that reprioritizes your search as you go along: "That looks like a flag because flags tend to look like that.", etc.

I have no doubt that the OP is good at reasoning logically, but the take-home lesson here is, if you want to be good at debugging, do a lot of debugging.

[danso]

Yes, this is true, so as I said, debugging looks easier in hindsight, especially someone else's hindsight :)

Summarizing the reasoning process:

1. The program's object space doesn't contain an absurd number of small objects, so inspect the core dump

2. 95% of the core dump is leaked objects, so a random sample should contain clues to the composition of the leaked objects.

3. A repeated pattern in every leaked object indicates a common pointer, i.e. a single type of object.

4. The signature helps find what file in the program is being referenced, which indicates that the pointer's object type is a BIO struct

5. This kind of leak isn't possible in straight Ruby. So between OpenSSL and EventMachine's C/C++ code, the latter is more likely to have something awry.

6. Search the EM code for BIO constructors

7. Check each constructor call to see if the BIO instance is ultimately freed

---

Steps 3 and 4 are the most arcane to me...I am pretty sure I do not know enough about memory to look at a hex signature and realize where in the address space it refers to, or even the significance of it.

[spoiler]

I am familiar with step 3, but I don't even know where to begin with step 4. I guess it'll remain a mystery... 4ever.

[to3m]

The reasoning is reasonably well explained in the article, I think?

Noting there is an identical pointer in each object, assume this is a pointer to some kind of type definition struct. What will this have in it? There'll likely be a type name, some function pointers for standard operations, maybe a link to any more definitions, that kind of thing. This kind of arrangement is just something you do in C and C++. It's very common.

So take a look at what's at that address, under the assumption it's just a block of pointers. And then look at the first pointer (0x401 is obviously not a pointer). And in this case he got lucky, because it points to a suggestive string. (It could just as easily have pointed to another type definition, or something random. Though it's not uncommon for descriptors to have the name in the first field or two.)

Object type names are usually string literals so it's likely to point into the data segment of an EXE or DLL. (Maybe on Linux you can guess that from the address as well.) That's what the maps thing is all about - figuring out where the string might come from as a way of narrowing down the search.

[icambron]

Yup. Wasn't meaning to disagree with anything you said. Just putting a different emphasis on it than you did.