[CamperBob2]

Honestly, if I can't have the whole thing, I'm not going to bother mirroring a 1TB fragment that's worthless by itself to everybody except copyright attorneys.

As ndriscoll points out, the only feasible way to distribute an archive of this size is with physical hard drives. I sure wish they would find a reasonably-trustworthy way to offer that.

[MrDrMcCoy]

Most of the books are bloated PDFs. I'm slowly working on a project to reliably convert PDF to DjVu, which on average yields a highly readable document that's 33% of the original size on disk. The project is proving difficult, as the tooling for DjVu is quite moldy now, and often needs to be manually reviewed to ensure the file remains readable. Pdf2djvu exists, but it's highly unreliable, and thus can't be used in bulk. Other ebook formats are XML-based and tend to be similarly bloated due to the overhead of the markup. It's a hard problem with so little in the way of good file format choices.

[CamperBob2]

That sounds like a pretty terrible idea, TBH. All of the best tooling is for PDFs, as you note, and storage will only get cheaper.

Ultimately that content is going to need to be represented as raw UTF-8 text and encoded images, so I don't see much upside to migrating it from one intermediate lossy file format to another.

[squigz]

You are never going to have a physical copy of the archive. It's nearly a petabyte in size.

[ranger_danger]

I know several datahoarders that have at least 1PB, also archive.org grows by that much at least every day

[squigz]

I assumed that GP was an average person who doesn't have a storage array sitting at home. I'm not really sure why the IA is relevant here

[CamperBob2]

1 PB of disk space would cost about $10K at this point in time. Not exactly unattainable. Looks like it would fit in a volume of space about the size of a standard refrigerator.

I'd be OK with both requirements.

[squigz]

It doesn't seem reasonable to me to suggest that an average person would spend $10,000+ (and the time to maintain it) on a pirate archive, hence my comment.

On the other hand, contributing a TB or two to a torrent swarm is much more feasible for most people.

In any case, if you're okay with that, you should do it. Please report back in 6 months with how it's going.

[CamperBob2]

In any case, if you're okay with that, you should do it. Please report back in 6 months with how it's going.

Point being, if I tried to torrent the whole thing, it probably would take 6 months, and would likely get me booted from my ISP and/or sued. I would much rather buy a set of hard drives with the contents already loaded. Or tapes, as userbinator suggests.

(And as for the hypothetical "average person" you keep citing, I don't see anyone meeting that description around here.)

[ranger_danger]

> It doesn't seem reasonable to me to suggest that an average person would spend $10,000+

You're right, and I was not trying to suggest that. I was merely disagreeing with "You are never going to" because I know there are people who are reading this who can and maybe will.

[userbinator]

1PB is well beyond the point at which a tape drive and a bunch of tapes will be cheaper than hard drives, and likely more reliable.

[account42]

For archival, yes. Not if you want to access the thing with any frequency.