It also appears to lag Ghostery and Adblock in efficacy [1]. Granted, AreWePrivateYet is "presented by Ghostery." But Ghostery does not necessarily show up at the top of the list every month (see November 2013). AreWePrivateYet also uses a Stanford study's methodology [2] for its open-source model [3].
Update: Ghostery appears to outperform Disconnect even when both are put in "block all trackers" mode [4].
For none of the extensions measured I had to go out of my way to set them in block all mode (mine is definitely still geeky compared to others -- it's more suited for the NoScript/RequestPolicy crowd).
I just re-installed all of them, and now I find they are all easier than ever to set up at install time. I had to tamper with Disconnect defaults months ago when I installed it, just like Ghostery and ABP. I appreciate it's no longer required.
If I adopt your arbitrary rule it would be pointless to do comparative measurements of privacy protecting power. I rather click once on a button at install time in a wizard (ABP, Ghostery) to be able to perform meaningful measurements than having to adopt a nonsensical methodology which would provide no useful information to end users.
Rather than dismiss the results, I think you should reproduce it and make a diff to find out what Disconnect doesn't block, and see how it can be improved. Maybe I will work toward finding more about this in the next benchmark.
Update: Ghostery appears to outperform Disconnect even when both are put in "block all trackers" mode [4].
[1] http://www.areweprivateyet.com
[2] http://cyberlaw.stanford.edu/blog/2011/09/tracking-trackers-...
[3] https://github.com/ghostery/areweprivateyet
[4] https://github.com/gorhill/httpswitchboard/wiki/Comparative-...