[kragen]

Torrents don't consist 90% of those things either. The largest audio torrent on http://thepiratebay.se/browse/100/0/5 is a sample library for use in creating your own songs. The largest video torrent on http://thepiratebay.se/browse/200/0/5 is a collection of the X-Files TV show; #2 is a collection of 50 classic movies (A Clockwork Orange, American Beauty, Annie Hall, Apocalypse Now, and so on) --- the movie equivalent of Nineteen Eighty-Four. The largest software torrents on http://thepiratebay.se/browse/300/0/5 are chess tablebases --- used for research into the game of chess --- Microsoft Windows 7, and auto repair software called Alldata. The top "other" torrent on http://thepiratebay.se/browse/600/0/5 is a preservation copy of GeoCities, which contained the full weirdness of the late-90s web in miniature, and after a duplicate of the same torrent, #2 is an archive of chemical journals.

I think what you're probably thinking of is not library shelves but library checkouts. And I think that you'll find that library checkouts did consist 90% of trashy paperbacks, the text equivalent of pop movies, back when people got that kind of stuff from libraries instead of online.

[GHFigs]

Would you judge the contents of a library by the size of the books?

[kragen]

...yes? I mean, how else would you do it?

[GHFigs]

I think you don't understand what you're saying.

If you were to judge the contents of a library by the size of the largest items on the shelves (exactly as you have done with torrents), you would come away with the mistaken impression that they consisted primarily of dictionaries and boxed sets of language learning CDs. In fact, these items represent a very small portion of the items in the catalog.

[kragen]

I agree that it's a crude measure, but I don't think the situation is quite as bad as you make out; your intuition about paper libraries is misleading you.

What's being counted as a single item here is not a single bound volume of a chemistry journal, nor the entire archive of Bioconjugate Chemistry, but rather the entire chemistry-journals wing of the library: 539 gibibytes, including 226 different journals. By comparison, the latest five items on http://webcache.googleusercontent.com/search?q=cache:http://... are 3.7MiB, 11.7MiB, 350MiB, 730MiB, and 260MiB; the chemistry-journals library is some 2000 times the size of the median of these and 120 000 times the size of the smallest, which happens to be a two-volume book called "Great Moments in Mathematics".

It turns out that when you have a power-law distribution crossing five orders of magnitude, like the one that characterizes file sizes, rather than the much narrower distribution that characterizes book sizes, you actually can get a useful approximation of the makeup of the total by looking at the makeup of only the largest items. It's surely not an unbiased estimator, but it's still a useful one.

Feel free to invest the work to do a better approximation.

[GHFigs]

We do not agree. I am not saying it's a crude measure, I'm saying you're measuring the wrong thing. File size is the wrong thing to measure. It doesn't matter what estimate of file sizes you can come up with, because file size is the wrong thing to measure.

Unless one is loading a moving van or trying to estimate the number of shelves required to store it, one characterizes the contents of a library by the items in the catalog and their subject matter, not by the volume they consume. You don't go in and ask for "a cubic foot of books" any more than you torrent "a megabyte of music".