[sixtyfourbits]

Fortunately someone had the foresight to make a backup:

https://news.ycombinator.com/item?id=32972923

Z-library were in it for commercial gain (you could access a certain number of books but to get more you had to pay for a subscription). They started out as a fork of library genesis, whose mission has always been strictly non-commercial and about providing free access to everyone without limits.

Hopefully this will encourage more people to go back to the original libgen. I suspect Z-library's popularity was because of a better interface and larger collection, but I think lots of people didn't realize libgen offers all its books/papers for free without limits.

[pilimi_anna]

Holy shit! Did not see this coming, but glad we saved this collection just in time. (I'm Anna, the one who made the backup).

We are working on hosting this collection, as well as saving other large collections. Please consider supporting us, donation details at http://pilimi.org (we'll set up a patreon-like system soon)

[areoform]

Thank you so much for your efforts!

I have steadily become convinced that we should make a multi-billion year backup of all of humanity's knowledge. All of it.

Something that,

  - is resilient (can survive a nuclear explosion)

  - requires no power

  - doesn't require software to read/reboot from

  - (theoretically) lasts for at least 1 billion years

Copying from the Long Now Foundation's projects, I think we can achieve these goals by miniaturizing pages and etching them on some metallic surface (a titanium alloy), depositing a layer of some resilient transparent material onto this surface, and creating multiple copies.

A few copies for Earth. 2 or 3 for the Moon. And a few sent out of the solar system on probes like Voyager.

Voyager itself is a great example of what we could achieve. The golden records were made out of stable, inert materials and Voyager’s trajectory doesn’t intersect with any known object for billions of years. The records themselves will be intact for at least two billion years according to one estimate. They are, for all intents and purposes, functionally immortal parcels of information.

https://www.space.com/predicting-voyager-golden-records-dist...

Some simple math, if the pages could fit inside of a 10mm x 10mm square, then for a plate that's about the size of an average coffee table at 2' x 4', we could fit 7,432 pages.

Assuming that we have 50 billion pages, we'd need about 6.7 million such plates to fit all of human knowledge, so far.

It sounds crazy, but assuming we could get net costs per plate down to $500, each copy would be about $35M. Or, ~0.14% of an Uber. Alternatively, 0.002% of the F-35 program.

That's doable!

-

6.7 million plates will probably weigh a lot. So off-world copies might need to use an alternative encoding scheme.

Another problem is likely to be organization of the plates/copies. The hardest part might be putting it all together in a way that can be trivially decoded by human descendants, even if they don't speak our language or share our subspecies.

(apologies for any typos, it's very late at my end)

[anyfoo]

I think what you consider the hardest part, putting it together so that someone who does not speak any human language today deciphers it, is actually one of the least of your problems. Unless you deliberately encrypt the information (which you wouldn’t), I have a feeling that even Wikipedia alone might be sufficient for a motivated civilization to figure it out. Linguists and archeologists in our time have to do with far, far less, and they have reasonable success. Add in a few things like dictionaries, textbook, novels, and I have little doubt that it’s a big obstacle.

Rather I think you vastly underestimate what a billion years can do. The earth itself, and all that was on it, was formed a “few” billion years ago.

Our rivers alone have carved entire valleys into mountain ranges in much, much less time. I doubt a titanium alloy and some unspecified sort of super epoxy stand a chance.

And constant custody with regular restoration cannot be guaranteed for billions of years either.

That’s a massive problem for one plate (and its many copies) alone, more so for millions of unique plates…

The Long Now Foundation is only shooting for 10000 years, as far as I know.

[prox]

Off world would help, no atmosphere on the moon for instance makes for less deterioration, and a cave system would shield it from radiation.

On earth clay tablets have done a great job : we have tablets 6000 years old, so we know that works. That’s 60% of 10k already. Titanium seems expensive and might be melted down in time of need, like bronze has been often in the past. Clay tablets survived partly because it’s a ‘worthless’ material.

[Kuinox]

You need to bury it on the moon. There is a reason the moon is full of holes.

[hanniabu]

> On earth clay tablets have done a great job

The ones that survived

[prox]

Yeah, but they didn’t even try preserving it! Lots were found in average storerooms and so on.

[wikfwikf]

It is 60% of 10k, and 0.0006% of a billion years.

[GTP]

But can you engrave them with the precision required for miniaturization?

[GTP]

As for normal backups, maybe it is better to have a chain of medium term solutions for storage rather than aiming for a single long term one.

[OOPMan]

A multi-billion year backup is probably pointless.

The earth will be uninhabitable in that time frame due to changes in the atmosphere and beyond that the sun itself will complete it's lifecycle.

I know there's a big fad to "just believe" in a SF future that spans space and time but physical realities in this area are pretty rough.

I think the best thing people could do is realise that, eventually, everything ends and believing otherwise when it comes to the human race is much like believing in an afterlife.

[areoform]

Our survival is not the point. It is an act of hope and the manifestation of our goodwill to the rest of the universe. There is nothing more valuable that we can offer to the Universe than our culture, history, knowledge, and the Earth's biological data.

Imagine if you were an alien species who somehow comes across this capsule hundreds of millions or billions of years from now. It's proof of sentient life elsewhere! But then you date the U-238 and realize that they're probably all dead...

But, they've left all of their civilization, culture, heritage, and knowledge behind. And you get to experience that, even recreate a tiny simulacrum of their world. And it gives you something, it's a tangible form of communication and cooperation across aeons.

-

In the short term, the backup is probably useful to have. Imagine if a collection is lost to fire or some other catastrophe, and one of the closer ones could be used to bring it back. Case in point, the fire at Notre-Dame. In 2015, Dr. Andrew Tallon, an art historian, painstakingly scanned all of Notre Dame, https://www.forbes.com/sites/trevornace/2019/04/16/we-have-b...

I think it's important to be honest. Would you have flagged his work as pointless in 2015? After all, the Cathedral has stood for centuries, been photographed so many times, what's the point of a 3D scan?

After 2019's fire, those scans became important (unsure to what degree) to the restoration effort. At some level, by capturing and preserving history, Andrew Tallon helped save history. And now it becomes a part of the story.

That's the goal. To engage in an act of optimism for the betterment of us and all of humanity.

[literalAardvark]

An alien civilization knowing us just by an archived version of Horizon Worlds feels like a fate worse than death.

[knaekhoved]

Doomer fatalism is nearly as dumb as sci-fi optimism. It's 100% physically plausible, using technology we have today, to make humanity star-faring. The costs would be exorbitant at this point, but within the reach of human productive capacity (assuming we're willing to ditch the partial nuclear test ban).

[TheOtherHobbes]

The costs are not the problem.

We can barely get to the Moon and have no realistic prospect of making a self-sustaining Mars colony any time soon.

How is it credible to claim we can build starships that can cross lightyears without technological or sociopolitical problems?

The engines - which we're nowhere close to creating - are the easy part.

[loup-vaillant]

Getting to the moon only takes another JFK. We have the tech, we have the people, all we lack is the budget and a 5-10 year deadline. Also, another hard part is getting there safely enough. A 5% chance of dying out there is probably not acceptable nowadays. We probably need to go below 0.1% to attempt it again, and that's not trivial, especially with the possibility of solar flares beyond the magnetic protection of the Earth.

The real question is, is it worth making it a priority?

[knaekhoved]

I was responding to "physical realities in this area are pretty rough" - obviously there are other blockers

[411111111111111]

> It's 100% physically plausible, using technology we have today, to make humanity star-faring.

That kind of claim needs an actual source, because it's not.

The distances are so many orders of magnitude too high for any of today's technologies, making this a pure daydream.

The best we could plausibly do is travel within Sol, and that's not starfaring.

It would have to be a multi generational journey in which any refueling/restocking is impossible and any slip up would cause a full wipeout.

Starfaring is far beyond us... and that's unlikely to ever change for humans with a biological body.

[knaekhoved]

https://en.m.wikipedia.org/wiki/Nuclear_pulse_propulsion

[dkjaudyeqooe]

> The earth will be uninhabitable in that time frame due to changes in the atmosphere and beyond that the sun itself will complete its lifecycle.

At that point someone can just put it on a thumb drive and take it with.

It's not beyond the realm of possibility that people ultimately inhabit a spaceship that wanders the universe looking for a new home.

[dniculescu]

"It's not beyond the realm of possibility that people ultimately inhabit a spaceship that wanders the universe looking for a new home."

We are doing that right now.

[pmontra]

> 6.7 million plates will probably weigh a lot. So off-world copies might need to use an alternative encoding scheme.

For the Moon and Mars, carve them into stone there. You'll have to defend against meteor strikes so they'll be carved and stored deep underground. No idea about seismic activity on those bodies.

For outer space, maybe it's pointless because finding, boarding, unloading an interstellar probe and sending the cargo back home is not easy unless you have very advanced interstellar ships. An orbit around the sun could be an easier place to spot. The Library of Ceres or the Library of the Troians?

[krisoft]

> they'll be carved and stored deep underground.

But of course you need some marker then on the ground so people know where to look for it later if they want to read the data. Maybe some artwork? A giant monolith perhaps?

[ASalazarMX]

A giant pyramid, they can last thousands of years even in our aggressive environment, much more in the weak Martian weather.

[krisoft]

I love that. Should we dig in the Cydonia region on Mars? :)

Perhaps at the D&M pyramid?

https://www.jpl.nasa.gov/images/pia04745-the-cydonia-dm-pyra...

[akie]

Not sure if serious or sarcasm.

[burnished]

What about that seemed sarcastic? It read as a practical solution to an unlikely problem.

Orbiting the sun seems like a pretty stellar solution to the whole "keep something close and visible but safe without maintenance" problem.

Plus, think of how exciting it would be for a future civilization to discover that an object orbiting the sun was not a lump of rock!

[adhesive_wombat]

6.5 million plates at $500 each is 3 billion, not 35 million.

Just don't make them look gold (even if they're not solid gold), or they'll get looted and melted.

Though it would be fun if a murderous warlord ended up with a gotten crown made of the Wikipedia article for toilet paper orientation. [1]

[1]: https://en.m.wikipedia.org/wiki/Toilet_paper_orientation

[areoform]

Oops! I did a rough calc in my head but forgot to add in the two zeros. But it's still fairly "cheap." 0.14 Ubers (14%), or 0.002 Pentagon Monetary Black Holes (0.2% of the F-35 program)

[wengo314]

i think it's a brave assumption that gold will forever be as valuable as now.

[sokoloff]

It’s not that it would need to be as valuable forever but rather the opposite: that it be forever not looted.

[hanniabu]

Gold is a really useful material

[GTP]

Indeed, it is a reasonable assumption that it will get more valuable over time, as it is a scarce resource.

[literalAardvark]

Gold is extremely common in the solar system.

In the next few hundred years it's pretty likely that gold price will be marginal, as asteroid miners looking for less useless materials such as nickel, cobalt and platinum mine an excess.

So no, unless for some hidden reason asteroid mining doesn't work out, which is pretty unlikely given how badly the developed world needs that material.

[hacknewslogin]

Instead of metal plates with a clear coat, maybe laser etching diamonds? a 5cm square of diamond can store 25 exabytes of data. They will weight less then metal as well. https://newatlas.com/electronics/2-inch-diamond-wafers-quant...

[Shared404]

Wouldn't that fail this requirement though?

> The hardest part might be putting it all together in a way that can be trivially decoded by human descendants, even if they don't speak our language or share our subspecies.

[dredmorbius]

Diamond can burn.

Though other crystals should be highly stable. Quartz seems geologically and chemically stable, plain old glass ain't bad. Something reasonably cheap is probably preferable, both from the cost basis (an expensive-to-create archive is a challenge) and the repurposing challenge (a diamond-etched bibliographic archive might have other appeals to those who chance across it).

Even parchment proved sufficiently valuable that works were often repurposed (and lost) through palimpsests.

[pdimitar]

Holy shit, this reminds me of the Jedi / Sith holocrons.

[fsckboy]

> a 5cm square of diamond can store 25 exabytes of data

isn't the same idea viable if applied to silicon crystals (which we already grow lots of)?

[pilimi_anna]

https://archmission.org is doing something like this. It's a super cool project, and they work with the Long Now Foundation, the Internet Archive, etc.

[survirtual]

Just use this:

https://en.wikipedia.org/wiki/5D_optical_data_storage

5d is a bit dramatic but it is a femtosecond laser writing of quartz crystal that should, in theory, be stable for billions of years and can hold hundreds of terabytes.

Seems to be just what is needed.

[jl6]

Sadly it appears to be early-stage R&D vaporware.

[xerox13ster]

It was early stage R&D vaproware in 2013 when I read about it the first time.

https://www.scientificamerican.com/article/data-saved-quartz...

[survirtual]

That’s a shame. I saw it earlier on Reddit and read the wikipedia article, seemed promising.

[literalAardvark]

Microsoft's "Project Silica" is in early production.

[gilded-lilly]

Why not encode them microscopically, using some kind of easily decipherable code?

The main issue would be loss of language. You’d end up with a gigantic Rosetta Stone of indecipherable gibberish.

Better would be to convert all literature into hieroglyphics, or graphic novels (or both), and then etch them microscopically.

[areoform]

This size will have microscopic features! There is probably a balance between ease of scanning/technological sophistication required to see the pages, cost of etching, and the number of plates overall.

The Long Now Foundation has extensively studied the language loss problem and arrived at a fairly elegant set of solutions.

https://longnow.org/ideas/02022/01/20/linguistic-data-in-the...

[6510]

Encode in dna and do roaches with random pages on their back.

[unfolding]

1 billion years is a long time. would a metallic alloy last a billion years? maybe we are better carving it into stone

[burnished]

You'd want to select it special to be non-reactive, and even then limit or eliminate exposure to UV, oxygen, and temperature changes. Stone is also a changing material on this time scale and I think storage preference would largely be the same. I think one of the more salient differences is feature size; easier to make very small yet still legible images in metal than in stone.

[askiiart]

To be useful after such a long period of time, we'd also need to include materials on learning the languages.

[0xbadcafebee]

> I have steadily become convinced that we should make a multi-billion year backup of all of humanity's knowledge. All of it.

And I'm over here hoping humanity extinguishes itself ASAP so the rest of life can thrive before we kill it all.

Intelligent life seems highly overrated.

[pilimi_anna]

Note that their TOR domains are still up: http://zlibrary24tuxziyiyfr7zd46ytefdqbqd2axkmxm4o5374ptpc52... and http://loginzlib2vrak5zzpcocc3ouizykn6k5qecgj2tzlnab5wcbqhem...

[dkjaudyeqooe]

Those domains are up, but downloading doesn't work right now, possibly because their CDN uses non-onion domain names. They're timing out.

[renonce]

You can use z-lib for the index and download the files on libgen

[Normille]

If you use Telegram, you can use the Zlib bot to search for and get books sent to you over Telegram:

https://t.me/firstlibrarybot

[Max size <50MB]

[terrycody]

Z-lib has much more books than libgen, sadly

[dkjaudyeqooe]

A work around for PDFs it to select "Open in Browser" from the download popup menu.

[SalimoS]

Download is working for me

[charondust]

yes, and you can try to send the link to the email first and then download it in the email. It works.

[terrycody]

Sorry can you direct the how to method to us?

[dkjaudyeqooe]

Can confirm, but has a 20MB (per book) limit.

[lgoex11]

They stopped being active.

[dkjaudyeqooe]

Still working for me.

[pilimi_anna]

The Library Genesis forum is also down, but more likely because of overwhelming interest right now? https://forum.mhut.org/

[IceWreck]

Hey, why not push the collection (the extra books) back to libgen

[DoItToMe81]

If you're asking why Zlibrary didn't, it's because they got greedy and wanted to soak up donations that might go to Libgen by piggybacking off of their servers and having individual uploads for their own.

[giancarlostoro]

This was my first question honestly and involves a lot less effort.

Additionally working on a UI for libgen that is far better as an open source project might be worthwhile.

[voldacar]

How do you actually host something like this without being shut down, arrested, etc? Do you pay your server provider in bitcoin or something?

[eru]

You could host your stuff via tor? And have people do most of the downloading via bittorent?

[harias]

Yes and you host out of someplace like Russia where the US publishers can't get you.

[pilimi_anna]

The Author's Guild and the Association of American Publishers made some handy lists for people who need Z-Library alternatives!

https://www.regulations.gov/comment/USTR-2022-0010-0024 https://www.regulations.gov/comment/USTR-2022-0010-0015

[MVf4l]

Hi Anna, could you please confirm that your server and yourself are not under US jurisdiction?

[pilimi_anna]

I will make no claims about my identity one way or another. If you are worried for me, don't be. If you are worried for yourself, then please take actions in line with your risk tolerance.

[psychphysic]

More likely they are worried about throwing their money away if the US now has a taste for takedowns.

[vanderZwan]

Anna deserves/needs the support just as much if they have to deal with US takedowns though

[psychphysic]

I disagree the community should get behind the most viable mechanisms to achieve its goal.

Hosting in the US, by someone in the US is foolish if your goal is breaking US copyright laws.

[beanjuiceII]

thats good because i've already forwarded this to my superiors

[wholinator2]

Maybe a naive question but assuming this isn't a troll, I'm curious why?

[beanjuiceII]

its not a troll, just my job, is your question why am i doing my job?

[pelasaco]

thank you for your work!

> Z-library were in it for commercial gain (you could access a certain number of books but to get more you had to pay for a subscription). They started out as a fork of library genesis, whose mission has always been strictly non-commercial and about providing free access to everyone without limits.

What is the difference between what "z-library" was doing, which was "you download a lot for free, but if you need more, just pay us, because no internet bandwith is for free" from donations? I always understood the subscription at z-library as support, not different than patreon...

[xwowsersx]

Forgive me, but I'm unclear on how I access the collection. I can just go to the torrent files as listed here: http://pilimi.org/zlib-downloads.html ? Is there a mirror where I can search for a particular book?

[mvkg]

You mirror Z-Library but don't support TLS on any of your sites. This seems like an odd choice. What is your threat model?

[primax]

Probably 'I want to archive books'. Not 'I want end to end privacy for anyone clicking on my plain text site with a few torrent links.

[chaxor]

TLS doesn't matter / isn't needed here right? The IP address is still in the header from what I understand. So the only thing https can hide is the content, such as a credit card or password that you enter into the site. The fact that it's a plaintext website that doesn't change means that the exact same information is encoded in simply giving the IP address as there is in knowing that someone looked around the site - because there's nothing else to do? I would like to learn more about how I am wrong, if I am wrong.

[cesarb]

> So the only thing https can hide is the content, such as a credit card or password that you enter into the site.

TLS is not only for hiding the content, it's also for authentication: it ensures that no malicious middle party can modify the content, for instance to inject malicious Javascript (for an example of this happening, read about the "Great Cannon" attack on GitHub).

[candreasen]

It also hides what URLs you visited. Depending how the hosting is implemented, just being able to see the IP, or even the domain name, wouldn't show what you were doing, but if it's in plaintext, they can see exactly what pages were visited and files downloaded

[elashri]

Care to add HTTPS support also?

[ijkxyz]

Do you backup user accounts and bookmarks too?

[pilimi_anna]

Sadly not yet.

[joshxyz]

thank you for your work

[terrycody]

good job anna!

[__jambo]

thankyou!

[dghlsakjg]

I'm not so sure there was commercial intent.

Their policy was basically around discouraging datahoarders and scrapers as far as I could tell. If I recall correctly, the limit was something like 10 e-books per day per ip address. Not really a huge limit when you consider that there was no limit on file size for things like graphic novels. They never required accounts, and the only thing locked behind a required donation was compute intense tasks like conversion, and send-to-kindle.

The kind of bandwidth/storage that they had to be consuming is quite expensive and I don't blame the organizers for soliciting donations.

[ezplorer]

You're right. I don't know why there's people repeating that "they have commercial intent" when it's actually not true. That's unfair, ZLib is a really good web

[meltedcapacitor]

They used various dark pattern marketing techniques to try and convince people to pay and make the site look like a legit library, so clearly some commercial intent even if it was always possible to download for free (maybe to provide some deniability?).

[dghlsakjg]

I disagree.

The only thing approaching a dark pattern is asking you to sign in for advanced features. A free account gets you a bunch of those features.

The main page has a single 'donate' link in the upper right corner. A contribution as low $1usd was enough to unlock all features except higher daily download limits.

What dark patterns did you see?

[meltedcapacitor]

- Making the service look like a legit library instead of clearly saying it's a repository of stolen books. Was confused myself for a while when I first stumbled on it, had to infer from it being too good to be true and doing some due diligence search.

- Various tricks to get the user to give an email (even if not necessary, similar to cookie boxes making "accept everything" very much easier than "only essentials".)

- After some time (download count or delay?) spam emails to try and get some payments out of the punter, again under false pretence.

Not saying it was bad value for money. Possibly not much worse than tricks average legal businesses employ. The point is someone sat down and devised that part of the UX with no other purpose than extract more money from users, by lying to them.

[dghlsakjg]

Fair enough.

It never occurred to me that people wouldn’t realize that it was a piracy site, and not a “library”. I guess I’m just too tuned into the internet. Haha

I see your point now. It’s sort of disappointing that I'm so used to even worse dark patterns on legit sites that I don’t really see this as dark.

[yencabulator]

> 10 e-books per day per ip address

That's a silly policy in a world of shared VPN endpoints and Tor exit nodes.

[bawolff]

> Z-library were in it for commercial gain (you could access a certain number of books but to get more you had to pay for a subscription). They started out as a fork of library genesis, whose mission has always been strictly non-commercial and about providing free access to everyone without limits.

Ick.

I'm sympathetic to piracy, but the moment people start to make money off of pirated works it starts to feel much more wrong.

[dghlsakjg]

The download limit was per day per ip address, and they would remove limits for active community members. I have a hard time believing they were doing anything more than paying for hosting bills.

[ezplorer]

They were not.

>Z-library were in it for commercial gain (you could access a certain number of books but to get more you had to pay for a subscription)<

That's a lie mixed with some truth. They have a download limit yes, but one that resets every 24 hours. You can download free 5 books even without an account, and with an account 10 and even more if you used the telegram bot. Like other user mentioned, that policy probaly was to discourage datahoarders and scrapers. And if you donated to the project you recieved the benefit of being able to download more books per day.

[flyingsky]

> [...] but the moment people start to make money off of pirated works it starts to feel much more wrong.

"Piracy" is expensive! Most people do not donate to support these things it's understandable if they decide to charge a few cents or bucks to keep the light on.

[acomjean]

Are you suggesting people who pirate don’t want to pay?

[braingenious]

Well damn, I guess I’ve gotta figure out how to get 30+TB of storage. Does anybody have any suggestions for doing that as cheaply as possible?

[tttttt5ts]

Buy 3x 16 or 18 TB external harddrives on black Friday and shuck them. It is still going to run you $400-$500. Can't do it cheaper as far as I know.

[bl4ckneon]

Yep I have done that before to make a 6 bay NAS. Gotta tape over a couple connectors on the sata connector but works great. Datahorders on reddit is a great sub for that sort of stuff. Guides as well as people posting when the deals happen for WD elements (external hdds)

[mister_tee]

not an argument but just an FYI for people reading about that idea for the first time: some NAS enclosures (QNAP, Synology?) and some power supplies can use the drives directly without disabling that 3.3V pin.

[zaarn]

Most enterprise level stuff can (ie, anything that'll go into a rack, the expensive QNAPs/Synos, any supermicro I've encountered, etc.) handle the 3V3 pin issue just fine, in my experience.

[flaviut]

Black Friday isn't always the best time: https://camelcamelcamel.com/product/B08KTRBHP1

But great advice and the route I've personally taken :)

[flutas]

Best source I've found is BestBuy with Easystore drives.

In fact they have 14TBs for $199.99 right now ($14.28/TB)

https://www.bestbuy.com/site/wd-easystore-14tb-external-usb-...

[tttttt5ts]

Yep, just predictably has some deals. eBay can be a reasonable option too.

[OmarAssadi]

It'll depend on where you live and what your goals are. If you have free-time to tinker and enjoy that kind of thing, you can build something very fast and reliable and prevent e-waste by building your own storage server with used parts on the cheap.

If you're in the United States, electricity is cheap enough that you can pick up much older SAS drives for really low $/TB cost and have it be worthwhile.

For example, I bought a used Supermicro CSE-836 [1], which is like a 3U server chassis with 16 hot-swappable drive bays and a backplane of some sort.

The backplanes vary, but mine came with the BPN-SAS2-836EL1. I paid $300 in total for the chassis itself, backplane, dual power supplies, heatsinks, etc, along with a Supermicro X9DRi-LN4F+ [2] and two Xeon E5 2660 V2s as a bundle from someone in the 'ServeTheHome' classifieds section [3]. From there, I picked up a load of HGST 3TB 7200rpm SAS2 drives on eBay for about $10 each from a recycling company. And then 192GB of DDR3 ECC memory from the same place for about $80.

I also grabbed a couple less-than-production-ready 3.84TB U.2 NVMe drives on eBay for a little over $100 each.

I think if I were to do it again, I'd have gotten slightly larger, newer drives. These are all totally fine, but I started seeing ~6TB drives for about 3x the cost per terabyte, which would pay itself off quickly with the energy reduction. The other reason is that I ended up going a little overboard; I have about 56x3TB drives right now, which is a lot more than 16, so I needed to get a couple of JBOD expansions to put them in, each of which were like $250 -- if I had gotten fewer, larger drives, I'd have had another $500 to work with & be saving on energy.

Another thing I'd have done differently is get fewer but larger sticks of memory. I have a really nice amount of RAM right now, but the energy consumption with 24x8GB isn't worth the upfront savings compared to getting 16 or 32GB DIMMs.

All the storage is in OpenZFS on Linux. The 56x3TB drives are configured as 7 RAIDZ2 vdevs, so 2 drives each are for redundancy, and 6 for actual usable storage. This leaves me with a bit over 100TB of usable space. And the 3.84TB U.2 drives are mirrored and act as a "special" device (lol, literally what they are called) [4] to automatically store small blocks and ZFS metadata.

I am sure I could have done a bunch better, but, so far, everything has been lightning fast and reliable.

I am using ZFSBootMenu [5] as my bootloader. It's cool since it is basically a tiny Linux distro that lives in your EFI and comes with a recent version of ZFS, so you can store your entire OS, including your actual kernel and such in ZFS, and you can enable all sorts of ZFS features that GRUB doesn't support, etc.

This is nice because, since the entire OS is living in ZFS, when I take snapshots, it is always of a bootable, working state, and ZFSBootMenu lets me roll-back to a selected snapshot from within the bootloader.

The Supermicro board has a slot for a SATA DOM [6], which is sort of like the form fact of an SD card. I picked up the smallest, cheapest one I could on eBay for like $15 and use that to store my bootloader. I did this so that my tiny 128GB SSDs that I use for my OS could be given to ZFS directly for simplicity instead of having to carve out a small boot partition, etc.

All in all, I'm probably out about $1750 for >100TB usable, redundant, fast storage, and a decent bit of power for virtualization and whatever else. It costs me like $50ish a month in electricity because of all the drives and DIMMs. But I was already paying 65 euros a month for a 4x8TB server from LeaseWeb to use as a seedbox, and ran out of space, so it's been worth it, even with my dumb decision to use 3TB drives.

[1]: <https://www.supermicro.com/en/products/chassis/3u/836/sc836b...>

[2]: <https://www.supermicro.com/products/archive/motherboard/x9dr...>

[3]: <https://forums.servethehome.com/index.php?forums/for-sale-fo...>

[4]: <https://www.youtube.com/watch?v=QI4SnKAP6cQ>

[5]: <https://zfsbootmenu.org>

[6]: <https://www.supermicro.com/products/nfo/SATADOM.cfm>

---

Edit: Also, figured it'd be worth mentioning, but the way I got the chassis+motherboard+cpu bundle for such a decent price was by posting my own thread. So, if anyone reading this is broke like me and not finding anything suitable, that is an option.

You won't always find exactly what you're looking for if you just browse around. But I've always had good luck explaining my situation, my budget, my goals, and someone tends to have stuff they don't need.

eBay seems to be pretty useless right now for the chassises (chasses? chassi? I give up) due to memecoin Chia miners. Forums are your best bet if you don't want to pay scalper rates.

[E39M5S62]

I'm one of the two ZFSBootMenu authors - it's great seeing people using it in the wild.

I'm not sure if this is something you're doing already, but don't forget that if you zfs send a snapshot to your storage pool, you can boot from that snapshot in a pinch.

We also (this past week) added a modifier to zbm.prefer - !!. If you specify zbm.prefer=zroot!! on the KCL, it'll only import the zroot pool. This might help your boot times by skipping importing your 56 drive storage pool.

We should have a new release out in the next few weeks, otherwise that feature is in the master branch if you build your own EFI executable.

[OmarAssadi]

Oh, interesting, cool! I appreciate the advice -- I am definitely not making use of all of the features.

And thanks for the software! I've been using it on all of my non-Apple devices for a while now. And my mom has been using it on their computer for a fair bit as well. It's been a total lifesaver on a couple of occasions and streamlined a lot.

[jrumbut]

I really like your writeup and I like how you shared some lessons learned and other stuff that didn't quite work out. Buying hardware can be a real adventure and it's a skill that for some of us is underdeveloped.

I'd encourage you to write more of this technical content!

[OmarAssadi]

Thank you, really! Maybe I am just in a weird mood, but that made my day, as strange as it might sound.

I don't think people, in general, are anywhere near as vocal and explicit about the things they appreciate compared to the things they actively hate or that frustrate them. And I think this applies even more so in careers and hobbies like tech where people don't seem to think of the things you make as consumable or creative in the same way they do with others (e.g., movies, videos, music, and other art); my personal experience has been that if someone is reaching out, it's—more often than not—to let you know about an issue or complaint.

So, thanks, it felt nice! And thanks for the reminder; I should let others know the same more often too.

[pdimitar]

> And the 3.84TB U.2 drives are mirrored and act as a "special" device (lol, literally what they are called) [4] to automatically store small blocks and ZFS metadata.

Man, even enterprise servers, sold directly to data centers, max out these devices at 64 gigabytes. Many are offered with a pair of 32GB SLC old-school SSDs (for the extra durability).

Using 3.84TB for this is an enormous waste unless you host half the GitHub yourself and have specified "all files under this or that size go to the special devices".

Better make a torrent or book or comics mirror or something on this pair of huge SSDs. I use a pair of 32GB extra-durable (SLC) USB pen drives for special/metadata on a fairly decent ZFS dataset (~9TB) and they have something like 4MB of taken space...

[TheSpiceIsLife]

Can I ask what you pay per kWh?

Have you put a power meter on that setup? I've always wanted to do something like this but electricity here is AU$0.27/kWh, and my house is poorly oriented for solar panels unfortunately.

[OmarAssadi]

It's a little under $0.12 per kWh here. I haven't gotten around to putting up a meter on it since fully completing it, but in the past, with a bit over half the drives + all of the ram and everything else, it was peaking a bit over 400w -- I'd guess like +- 700ish right now?

So, give or take, with a stupid build like mine, here, it's like .7 * 720 * $0.118 = ~$60.

Another thing to note is that mine stay spinning instead of idling for much lower energy consumption, cause I've always read that the stop-start cycle is what really tends to kill the drives.

I think it could be a lot more affordable/justifiable with:

- fewer, larger drives (6-8TB each?)

- fewer, larger DIMMs (16-32GB each?)

- perhaps slower, more energy-efficient drives (5400rpm?)

Also, I have seen some of the Atom C2750/C2758 boards for fairly cheap here.

If you don't need an absurd amount of RAM/PCIe/CPU, that could be a good option to save another bit of power.

> and my house is poorly oriented for solar panels unfortunately.

And yeah, same, unfortunately. Ironically, despite being in Florida, I barely get any sun; my entire neighborhood is filled with massive trees, lol.

I'd still like to look into solar at some point when I've got more time & money, though. It's been awful the past couple of hurricane seasons because of the trees -- something always seems to fall and cause a bunch of damage, so I'd like to cut down a few of the scary ones anyway (I am sorry trees!)

[jvdvegt]

0.772012=6000kWh for a whole year. My household of four uses 4000kWh...

[crumpled]

If you can't seed the whole collection, just seed a random portion. You can still cherry pick what you like.

[snvzz]

IPFS would be a start.

Actually, aren't they already using that?

[fanick]

With obvious limitations you could use backblaze. Back of the napkin calculation this will cost you only $150 per month in storage fees.

[tuco86]

big HDDs are getting cheap. just checked, 16 tb is only 260E by now

[1ark]

4-5 Bay hard drive enclosure (with RAID)?

[boardwaalk]

RAID isn’t cool anymore. Maybe ZFS.

[zozbot234]

I wonder if there's any worthwhile public domain content in that backup? E.g. stuff published prior to 1925, that's not otherwise findable on the usual services (Gutenberg, IA etc.)

[roel_v]

0 seeders on the few torrents I checked though. I mean, can't blame people for not seeding 350GB+ torrents, but still - if there aren't any seeders, what good are the torrents?

[golem14]

[this is not legal advice]

Practically speaking, if you are in the US, you could download stuff from zlib and get away with it - I believe that's not illegal. Hosting on the other hand is.

Using torrents to download is legally also hosting, albeit some random parts of what you are downloading yourself. So to stay legal in the US, you can't really use torrents.

Maybe some enterprising person in Russia or China can become a reseller of hard drives and preload them with chunks of libgen ?

I mean, it's sad that the greatest country on earth has a sneakernet gap with Cuba [1]! ;)

[1] https://en.wikipedia.org/wiki/El_Paquete_Semanal

[lgoex11]

Z-library has books that Libgen doesn't have... I tested them.

[nephanth]

I thought only the domain name was seized though ? Couldn't one make a backup now bu connecting using ip address, or an alternatie DNS server

[robertwt7]

when I go to libgen though, link number 2/3 always point to zlib though.. am I on the right libgen? are they connected?