[Nican]

I like reading about IPFS, but I do not really have the time to learn about it and get involved.

Last I remember, Z-Library was having an issue scaling the DHS to handle the number of files [1]. Did those issues get resolved? How is it going now? Also, is there anything being done to ensure every file has seeders?

[1] https://news.ycombinator.com/item?id=33716560

[survirtual]

I’ve dived deep into IPFS and built several prototypes on top of it. It ended up not being performant enough for me, and that was after heavily modifying the codebase so that it was true p2p browser & server (their webrtc transport had a lot of issues and they didn’t seem too interested in it, but my needs required it as a backbone).

The security was also a concern, and the scaling had issues. Pinning millions of small items got so slow it would not function. Then I ended up having concerns over hashed based addressing being easy to censor with the architecture IPFS was using (more hub & spoke than anything, given signaling and relay servers were centralized).

I could go on but I ran into so many issues I ended up implementing my own solution that did everything I wanted. Wanted to squeeze even more performance, I’ve been converting that solution to Rust.

This was a couple years ago so maybe things have changed since I used it. Last I checked, they seemed busy on Filecoin.

The idea of IPFS is great and I want to see it succeed, but I think that they got so caught up in their jargon and modularity, the project lost track of some fundamentals.

[oldgregg]

Honestly I think Consensys/IPFS/Libp2p is just some corporatized way to derail real P2P and decentralization. Their libraries are total garbage. Lots of complicated code that simply doesn't work. No documentation. I mean look how much IPFS and Libp2p is pumped but IT DOESN'T WORK. IPNS is a joke. All way overengineered crap that does everything but actually nothing. Look at the $$$ and pedigree behind Consensys it's 100% establishment.

[survirtual]

Anything is possible, but having some limited github interactions with the core team, that seems unlikely. My impression is that they are a passionate group that hit the jackpot a bit prematurely.

Their intentions seemed good to me they just have an ungodly amount of financing while perhaps lacking a core vision & understanding of what is at stake.

Put another way, seems more like an academic building something rather than a seasoned industry pro.

In any case, I don’t want to disparage their project. I learned a lot from their code & concepts. We are all on our own roads towards brilliance, contributing to each other in all kinds of ways.

[oldgregg]

I certainly wouldn't disparage the talent- no doubt vast majority have good intentions but unfortunately they are being paid handsomely to have their work be rendered ineffective. Gobble up all the best P2P devs and get them focused on work that is largely ineffective. Lots of code and specs but nothing really usable. Code is not very original, they just take other libraries and tweak them. mplex is their version of yamux, autonat is their version of stun/ice... but then none of their libraries really work together seemlessly. Embrace extend extinguish? If you look at their massive code output then google "what apps use libp2p" you won't find anything because nobody uses it. Berty is really only app worth a mention.

[survirtual]

I’m not saying conspiracy doesn’t exist — it most certainly does — but it just seems unlikely in most cases.

In this case, to be blunt, give a bunch of nerds millions / billions of dollars and see how fast their objectives change. Going from being students to having material wealth unimaginable to most suddenly gets you invited to a lot of conferences / parties / inner circles. You’re doing a lot less engineering and a lot more socializing & experiencing, and suddenly those lofty ideals & nerdy curiosities may just not matter as much any more driving around in a lambo with what you’d consider super models. But making even MORE money to do cool things just might.

Very few people exist on this planet that can resist that, and it is the explanation for most people / projects / orgs that make it big losing their way. Once you have a lot of money, suddenly life without it seems impossible, and you start to compromise values to secure more of it. It is a steep downhill journey.

Most projects are unoriginal. Just a derivation slightly improved on, or a recombination of prior works in a compelling new way. Nothing wrong with that; our universe is composed of relatively few atoms combined in all kinds of special ways.

[dfghjkjhgfdx]

you two are discussing semantics :)

you both are correct on all assumptions. and are now discussing how corruption with money within the existing system is both undetectable/deniable and effective.

ipfs is now, for all purposes, a honeypot against dweb. or at least a way for opposition (dweb proponents) to waste valuable time. as your experiences proved.

[2color]

I think that this is a misunderstanding but I understand where this is coming from.

Let me share my 2 cents as the Developer Advocate for IPFS (employed by PL).

You claim that our work is largely ineffective but what are you referring to? The IPFS ecosystem is like a cluster of several ecosystems, e.g. Libp2p, IPLD, IPFS, Multiformats.

> Lots of code and specs but nothing really usable

Some of the libraries may not be as actively maintained as others — this is the nature of open source. But to claim nothing is usable is not accurate — there's a lot of stuff that works pretty reliably.

> Code is not very original

Who cares....

> autonat is their version of stun/ice.

It's actually Autonat, DCUtR, and Circuit Relay V2 which are pretty amazing given that it's like STUN/ICE but decentralized! Anyone in the network might help you hole punch through NAT and from network measurements the success rates average around 70% (https://youtu.be/bzL7Y1wYth8?t=1091). Saying it's just "their version of STUN/ICE) takes away from the main qualitative difference of doing it without centralized coordination servers.

> "what apps use libp2p" you won't find anything because nobody uses it.

Ethereum uses libp2p GossipSub since the beacon chain launched. Berty, Satellite (https://satellite.im/), Status (https://discuss.status.im/t/waku-version-2-pitch/1776#org900...).

Either way, I appreciate your comment, we need to be doing a better job of surfacing some of the great progress being made in the IPFS/Libp2p ecosystem.

[1vuio0pswjnm7]

Peer-to-peer is too general and too useful to be spoilt by "blockchain". But there is so much money behind this crap it is a formidable virus that can infect any project.

ConsenSys is a company formed by Ethereum co-founder. He has sold out to the big banks, in the opinion of some Consensys shareholders.

https://finance.yahoo.com/news/consensys-shareholders-readyi...

"Falls says he believes both MetaMask and Infura - what he calls "the crown jewels" of ConsenSys - could have been decentralized and tokenized, and that their projected use is "completely anathema" to the peer-to-peer principles of the space.

Meanwhile, a number of teams are looking to bring "institutional DeFi" - with its known counterparties and compliant custody arrangements - to the marketplace.

"Forget about the shareholders for a minute," Falls said. "Think about the consequences of the change in the influence over these infrastructure pieces.""

Out of curiosity I was looking at the evolution of IPFS hashes. They have gotten more complex. IPFS now uses base58btc exclusively. That's "btc" as in Bitcoin. Something like rhash, which has traditionally supported hashes used in peer-to-peer protocols, has no support for base58btc.

People are now trying to associate IPFS with "Web3". For example, check out this paper published a couple of weeks ago.

"Studying the workload of a fully decentralised Web3 system: IPFS"

https://arxiv.org/pdf/2212.07375v1.pdf

[2color]

> IPFS now uses base58btc exclusively

That's blatantly wrong. IPFS supports 25 different base representations (https://github.com/multiformats/multibase/blob/master/multib...).

In fact, recently, two community members decided to implement a new base encoding with emojis for fun:

https://cid.ipfs.tech/#%F0%9F%9A%80%F0%9F%AA%90%E2%AD%90%F0%...

https://github.com/multiformats/multihash supports at the very least SHA1 SHA2-256 SHA2-512 SHA3/Keccak Blake2b-256/Blake2b-512/Blake2s-128/Blake2s-256 Blake3 and Strobe. Hashes in IPFS are being standardised through the IETF and W3C https://www.ietf.org/id/draft-multiformats-multihash-05.html.

If you need rhash, you are welcome to submit a PR! We also have a grants program you can use to be rewarded for this.

[1vuio0pswjnm7]

More on financial relationship between IPFS (Protocol Labs) and Consensys (Ethereum co-founder):

https://consensys.net/blog/press-release/consensys-raises-65...

https://www.reuters.com/technology/blockchain-firm-consensys...

https://consensys.net/blog/press-release/protocol-labs-colla...

[1vuio0pswjnm7]

Above I stated, "IPFS now uses base58btc exclusively." A more precise statement would have been something like, "IPFS CIDs originally used base58btc exclusively." It looks like IPFS is moving to a new CID protocol that allows use of non-BTC hashes. That's a step in the right direction, IMHO.

"In CIDv0, hashes are always encoded with base58btc. Always."

https://web.archive.org/web/20200810110206/https://proto.sch...

"Converting CID versions

You can convert any CIDv0 to CIDv1, because the implicit prefixes from v0 become explicit in v1. However, because CIDv1 supports multiple codecs and multiple bases and CIDv0 does not, not all CIDv1 can be converted to CIDv0. In fact, only CIDv1 that have the following properties can be converted to CIDv0:

multibase = base58btc multicodec = dag-pb multihash-algorithm = sha2-256 multihash-length = 32 (32 bytes, equivalent to 256 bits)"

https://proto.school/anatomy-of-a-cid/06/

Thank you @2color for the correction.

[mhluongo]

Again, ConsenSys isn't behind IPFS. These are two different things.

> People are now trying to associate IPFS with "Web3". For example, check out this paper published a couple of weeks ago.

IPFS has been used to host dApps for a long time — eg the Uniswap frontend. This isn't a recent thing.

[RunSet]

> IPFS now uses base58btc exclusively. That's "btc" as in Bitcoin.

It is true that Base58 is named for its original implemention in bitcoin but other than the name it has nothing to do with bitcoin.

The difference from Base64: It excludes non-alphanumeric characters and visually similar characters (I, l; O, 0) to make the output more human-friendly[0].

The problem it attempts to solve: Making bitcoin addresses more human-readable. That is likely also why IPFS uses it.

[0] https://en.bitcoinwiki.org/wiki/Base58#How_it_works.3F

[mhluongo]

You're confusing ConsenSys and Protocol Labs.

ConsenSys was founded by Joe Lubin, and is the corporate entity behind MetaMask, Infura, etc. It counts MasterCard and JP Morgan as investors.

Protocol Labs was founded by Juan Benet and is the primary sponsor of IPFS and the dev team behind Filecoin. They're funded primarily from the private token sales of FIL.

Agree with the sibling comment that the focus on modularity and solving "all of the things" rather than focusing on adoption seems to be a pattern with Protocol Labs, though I'm a big fan of what they're trying to build.

[TEP_Kim_Il_Sung]

ConsenSys is to IPFS as Blockstream is to BTC

[mhluongo]

No, Protocol Labs it to IPFS as Blockstream is to BTC.

ConsenSys is an entirely different company focused on Ethereum, and neither invented nor maintain IPFS. That's Protocol Labs.

Protocol Labs didn't start out of ConsenSys either — in fact, I don't believe they have too much beneficial ownership in common at all. In addition to the separate founding stories already mentioned, you can read about how Protocol Labs was initially funded by Union Square.

[TEP_Kim_Il_Sung]

ConsenSys is to Blockstream as MetaMask, Infura, MasterCard and JP Morgan are to AXA

Protocol Labs, as you correctly identified, is not a stooge of global corporate interests.

[KirillPanov]

Same experience here.

The #1 lesson they failed to learn from Bittorrent and Tox: your DHT must be sessionless.

QUIC is not a solution to this particular problem. Although QUIC uses UDP, it still creates and manages sessions. This is incompatible with decentralized scaling. At the scale of Bittorrent and DNS you cannot manage this kind of session state without centralization.

[2color]

IPFS encompasses more than just a DHT: - representing content addressed data with hashes in CIDs - Discovering them (via the DHT) - Transfering (Bitswap, HTTP, Graphsync, sneakernet and more in development)

I'm not 100% certain what you mean by the DHT being sessionless.

> At the scale of Bittorrent and DNS you cannot manage this kind of session state without centralization.

You don't have to route content via the public DHT. Depending on what kind of data you're publishing, you can use a private DHT swarm, or delegated routing with HTTP.

> This is incompatible with decentralized scaling.

[anacrolix]

Absolutely. Too many people try to implement DHT on top of stateful connections, it doesn't scale. It must be as cheap as a single packet or two, with no context.

[jzm2k]

Could you elaborate on why it can't scale if it's stateful?

[KirillPanov]

That should probably read "doesn't scale without centralization".

The full answer to that question is way, way, way beyond the appropriate length for an HN comment.

DHTs need to accept requests from any random IP address, anywhere on the Internet. If you allocate state, you have to deal with resource exhaustion attacks. Dealing with them turns out to be way harder than anybody thinks when they first start down this path.

Cloudflare and Google can use stateful HTTP(S) because they are so massive. They can employ hordes of people to constantly tune DOS filters to block griefers before state allocation occurs, and lawyers to deal with any griefers who are as large as they are. If you're happy with that kind of centralization you shouldn't be wasting your time with DHTs; you don't need them.

Aside from resource exhaustion attacks, there are some pretty deep lessons connected to systems and architecture. The bittorrent bootstrap nodes serve some really insane number of requests/second on very modest hardware (I'm still trying to find the link that detailed this...). The Internet root zone DNS servers were, for a very long time (and perhaps still today in one or two cases) just a few good workstations at universities.

Sessionless protocols scale like crazy, on cheap hardware. But they are very hard to design correctly. If you have VC bucks, spend them on hardware and use sessionful protocols.

[survirtual]

You can restrict DHTs to accept well-formed requests and further restrict them using a web of trust. With a large enough network and an on onboarding path using nodes that have higher bandwidth, it can work to mitigate that vector I’d think.

It would be really hard to launch that attack with a DHT that implements appropriate pruning, drop, and block rules for bad behavior. All that can feed into a personal (& autonomous) web of trust. Any references for me to read more about this?

[password4321]

> my own solution that did everything I wanted

Your ideas are intriguing to me and I wish to subscribe to your newsletter!

(I really would appreciate any pointers to additional info on something that actually works...)

[survirtual]

If you find something, let me know. It would save me a lot of trouble.

Until then, what I’m building unfortunately doesn’t seem to have a peer. So I will keep at it. If it gets to a point where I believe it can be useful to others, I will share it freely.

[codetrotter]

> If it gets to a point where I believe it can be useful to others, I will share it freely.

It sounds very useful already.

I’d love to see the code even if it’s not in a quite working state yet.

[davidwritesbugs]

I'll second that - even if the code is currently in poor condition. IPFS sounds great but it has a bad reputation for performance/resource hogging but from your original comment that seems to be the least of its issues. I'd love to use something like that for my project.

[survirtual]

Trust me, when I can share I will, and gladly at that. Hopefully it will be useful and many other people take up the cause. After that I’m taking a rest-of-my-life adventure around the world’s most pristine wildernesses.

[klabb3]

I’m building a connectivity lib in Golang for relay-assisted p2p over tcp. The implementation is simple, uses only a single network connection when idle, and establishes connections very quickly and reliably. Downside is it’s not decentralized and it’s also not UDP, in case that’s your thing. Oh and BYO auth.

Please reach out if it fits your use case.

[oldgregg]

Contact info? I'm currently developing a p2p tcp library in Golang based off perlin-network/noise which is a very clean implementation but abandoned.

[klabb3]

Sounds great, definitely some overlap there.

Added to my user page to avoid creating a permanent record. I’ll reply with my personal.

[chezball]

@survirtual, would love to see what you used your own implementation for. Is it open source? Or a product we might be interested in?

[ComputerGuru]

ZFS (Zettabyte File System) is very much separate from Z-Library (and its alternative frontends like libgen).

[Nican]

Thanks. My brain crossed wires between Z-Library and IPFS. I updated my post.

[ComputerGuru]

Completely understandable!

[benneh]

IPFS is not the only option available. Swarm's been making a lot of progress.

Hackernoon released a blog post comparing the two projects.

https://hackernoon.com/whats-the-difference-between-ipfs-and...

[steponlego]

Works great, that's old news which is no longer current.