[ispolin]

Did anybody test the mice for physiological changes, and specifically reduced testosterone levels? In Matt Ridley's "The Red Queen", he mentions research about how low testosterone levels in both males and females cause this type of behavior.

Maybe there's an evolutionary adaptation that lowers testosterone production during times of high population, and in the artificial conditions of the experiment that adaptation exhibited itself too strongly?

That would also explain why mice taken out of the experimental environment didn't rehabilitate. There are other studies cited in "The Red Queen" that suggest that levels of testosterone are largely determined by the mother's environment and stress levels during gestation.