I quite agree that it's grown a bit gnarled and tangled with optimizations over the years.
It's interesting to compare the current version to the earliest available copy of the annotated source code, from version 1.1.1, seven years ago, in a simpler JavaScript age:
Then I had an easier time following the annotated version after understanding how the other functions were built on top of each other in the simpler syntax.
For me, I started by reading and understanding the raw commented source code from the first release (without the initialization wrapper and initially confusing module setup): http://cdn.rawgit.com/jashkenas/underscore/f4299d7427abc59bc...
Then I had an easier time following the annotated version after understanding how the other functions were built on top of each other in the simpler syntax.