[jzelinskie]

This update to the spec is a modest change that's largely a preemptive reaction to SHA1 being broken; large portions of BitTorrent are designed around the 20-byte length of a SHA1 checksum. They've decided to move forward with SHA256 truncated to 20 bytes to avoid incompatibilities with existing infrastructure such as the Mainline DHT.

Beyond the hashing algorithm, some important additions that were previously proposals without widespread use (e.g. merkle tree for hashing pieces) are becoming required. The focus has mostly been on optimizing latency for the P2P protocol and making sane improvements to the file spec. I feel like trackers were largely overlooked in this update, but I'm biased because I work on a popular tracker.

Ideally, BitTorrent would be broken down into separate specifications that could be used together or in separate systems: one for the file format and piece representation for sharing files, one for the P2P protocol, and one for discovery (trackers, DHTs). I want to believe that there would be far more interesting P2P projects if you could just lift robust primitives from BitTorrent.

[the8472]

The DHT BEPs specify a network that is only barely related to the bittorrent core protocol, they can already be used independently, and some people do.

> I feel like trackers were largely overlooked in this update, but I'm biased because I work on a popular tracker.

Yes, we did not pay much attention to trackers, but BEP52 basically seized the opportunity to do some incompatible changes we always wanted to do anyway (quite a few accumulated over the years), and there were no such open issues with the http tracker protocol.

[jzelinskie]

>...and there were no such open issues with the http tracker protocol

This is because the HTTP protocol is so much overhead that most trackers don't even really run it anymore. I think UDP being promoted to the spec would've been a step in the right direction. Modern trackers have a bunch of tricks like BEP34[0] to avoid getting pounded that would be great if every client conformed to.

I hope I'm not coming off as aggressive. I really appreciate this work and I'm really glad to see a spec revision. It's just as you said, there's been many years and many good improvements that I'd like to see made while there's still a change to break things.

[0]: http://www.bittorrent.org/beps/bep_0034.html

[the8472]

I still don't see why this would need to piggyback on a breaking change of the metadata file format. "Please use UDP by default" is fairly orthogonal to the metadata format and could be added to the spec at any point if we want to.

HTTP trackers are considered fine for medium-scale torrent deployments. UDP trackers were originally introduced to cope with the traffic caused by running an open tracker that manages 100k+ infohashes for the whole world.

Also, both BEP3 and 52 already forward-reference the tracker extensions (compact and UDP), so someone who writes a new bittorrent implementation should already be aware of them.

Maybe we could make it more clear that some BEPs are almost-mandatory.

[aninhumer]

>The DHT BEPs specify a network that is only barely related to the bittorrent core protocol

Yeah, if I remember correctly bittorrent DHT ultimately just maps 20 byte hashes to peer-lists (IP + port pairs). It's obviously designed to be convenient for bittorrent swarm discovery, but nothing about it limits it to bittorrent usage. Indeed, I'm surprised it's not more widely exploited for p2p bootstrapping.

[pdimitar]

More generic question, apologies if it feels inserted without relevance:

Did you guys talk with the IPFS team? Do both of you have a desire to start bringing both families of protocols and technologies closer together?

I feel in this age we must make de-fragmentation of efforts our topmost priority.

[predakanga]

How do you intend to handle incompatibilities in the tracker protocol that are introduced by BEP52?

In particular, I note that there's nothing in there regarding which infohash should be used in the tracker updates. Should traffic with v1/v2 clients be reported separately, or should it be consolidated under the v2 infohash?

[the8472]

hybrid torrents essentially consist of two swarms, so you announce twice.

[userbinator]

SHA256 truncated to 20 bytes

What are the security implications of doing this? It seems it wouldn't increase the strength beyond the original 160 bits, no? Was there anything preventing redesigning the protocol to use full 32-byte SHA256 hashes throughout?

[baby]

SHA-256 has a 256-bit output. Truncated to 20 bytes/160-bit you effectively reduce the collision resistance to 80-bit and the (second) pre-image resistance to 160-bit. (This mean for example that you have to do around 2^80 operations (hash computations) to find a collision.)

80-bit of collision resistance is usually the number accepted for legacy cryptosystems or for lightweight crypto. It's not great but it's not "too bad".

By removing 96 bits from the state you also prevent length extension attacks (which SHA-256 is vulnerable to, see [1]). Or rather, provide 96-bit of security against them. Which should be enough.

This is better than using SHA-1 because SHA-1 has "efficient" chosen-prefix algorithms to find collisions while SHA-2 currently does not.

Now if it were me I would have chosen a hash function like KangarooTwelve which is faster, provides parallelization for large inputs, allows you to customize the output length and has received a substantial amount of cryptanalysis.

[1]: https://cryptologie.net/article/417/how-did-length-extension...

[scrollaway]

The problem with SHA1 being broken is unrelated to the amount of bits in the hash, so the point of switching to sha256 isn't to "increase the strength beyond the original 160 bits" but to avoid Shattered and future potential attacks.

[the8472]

There are 2 uses of the hashes in bittorrent. a) integrity checking b) as unique identifier for the swarm.

The hash only gets truncated when used in places as unique identifier. When you start with a v2 magnet link or torrent file you get the full 32bytes hash, which means your integrity-checking is unaffected.

[nenreme]

You download 32 bytes hash and check it using 20 byte hash from the magnet link.

[the8472]

http://bittorrent.org/beps/bep_0009.html#magnet-uri-format

[snakeanus]

Is there any plan to allow base64, longer hashes and selecting the hash algorithm in the magnet uri format?

[the8472]

No such plans. What would be the motivation anyway? A different hash function for the infohash calculation wouldn't change anything since other parts of the protocol would still use sha256 (v2) or sha1 (v1). So if this is about security, you wouldn't be changing the weakest link.

Not to mention that this would not be compatible with other peers, there is no algorithm-negotiation in the network protocols, so how would they know which hash you're using? The magnet link is pretty much the far end of the process, you'd have to change a lot of other things first.

[sha2throwaway]

Actually you do get some added security, because it prevents the length extension attacks sha-2 and related hash functions have thanks to the Merkle-Damgard construction [0]. Specifically, by truncating the hash, the output no longer contains enough state to perform the attack.

[0] https://en.wikipedia.org/wiki/Length_extension_attack

[loup-vaillant]

There are better ways to prevent length extension attacks, such as choosing Blake2b. With the current scheme, only 12 bytes are missing, so Length extension attacks only get 96 bits of security…

Replacing SHA-1 with SHA-2, what are they thinking? Blake2 is faster and more secure than either.

[simias]

Nobody gets fired for choosing SHA-256. It's the de-facto standard these days, it's the most likely to have hardware acceleration etc... I think it's a pretty conservative choice.

Furthermore I don't think hash performance is very critical for bittorrent, after all it's generally I/O bound.

Could you even exploit length extension with bittorrent? Given that it cuts files in chunks of the same size it sounds pretty hard to sneak extra bytes in there.

[loup-vaillant]

> hardware acceleration

Ah, that would explain a lot. (Not in this particular case, but why it seems to be the default choice in many settings).

> Could you even exploit length extension with bittorrent?

No Idea. I was just responding to a general point.

[derefr]

They're probably thinking something about which hashes will get hardware-accelerated instructions provided on x86 and ARM.

[loup-vaillant]

Yeah, I keep forgetting about Hardware acceleration…

[snakeanus]

Length extension attacks are a non-issue for torrents.

[boyce]

I'd love to see a p2p wikipedia and a p2p twitter alternative

Moving bittorrent away from it's present image could be achieved by making p2p useful beyond bluray rips

[fps_doug]

I liked how the old Opera included a bittorrent client. It was the only browser that did, and felt like it was actually the way it was meant to be used. But nobody got it. You had warez kiddies left and right complaining how it sucked compared to Azureus and later µTorrent which had bazillions of features to tweak and max out their connection, or saying it was useless because their favorite ALT did not whitelist Opera.

But I just used it a lot when running bigger downloads like install discs for Linux distros, OpenOffice etc., and it made a difference when there was some major release and half of the plain old http mirrors were painfully slow or down entirely. Admittedly, that situation got a lot better compared to 10 years ago, but still I'm delighted by how natural it felt to use, since it seamlessly integrated with the browser's download manager. And you didn't have this "uh, I need to start an external program for this" kind of reluctant thought when you saw a website offered download via torrent. Today I just wonder if BT would have evolved differently if all browsers would have included a client.

[johnwaynedoe]

The Brave browser has a bittorrent add-on pre-installed. I've been toying with it as my daily driver for a couple weeks and have been really enjoying it. The project still needs work but I love what they're doing.

https://en.wikipedia.org/wiki/Brave_(web_browser) https://brave.com/

[boyce]

What's your job there

[johnwaynedoe]

I wish I worked there, I do miserable tier 1 tech support. The reason I am promoting their browser is because it has the exact functionality the parent comment was looking for. Its also a really neat project by a former Firefox dev. It's a browser made to inherently block ads and tracking . Its essentially seamlessly does what several add-ons do for me in Firefox. I'm just excited to see a browser take tracking ads, fingerprinting and other browser privacy seriously.

[boyce]

I think there's still potential to steer BT in a more exciting direction if there was the will to from those involved. More quality software projects using it to download by default. Bring the trust back, shake off the reputation for just being trojan-infested cracks.

There's plenty of things besides piracy people could be doing with torrents and related tech and it seems like such a waste of an idea. A Linux package manager, an open-source Acestream alternative, collaborative work on large scales

[wongarsu]

Updates for large software (mostly games) is often distributed via BT. But it's mostly invisible to the end user because the client just does it transparently.

[boyce]

Well there's a start (but the bittorrent people should shout about those sorts of uses)

[cesarb]

Also, the original Bittorrent client looked like an Internet Explorer file download dialog. The download experience was nearly identical: you clicked on a download link, and a download dialog with a progress bar opened up. Behind the scenes, the torrent file MIME type was registered to the Bittorrent client, so it was downloading the torrent file and launching the Bittorrent client, while a normal download showed the download dialog directly, but the user didn't have to be aware of these details.

So yes, it's the way it was originally meant to be used.

[Klathmon]

Personally I really truly believe that once you can make a torrent client in javascript that runs directly in the browser without any plugins or gateways/tunnels that you will see it explode.

Developers will code it into their download pages, decentralized systems like a p2p wikipedia will be possible and always accessible by anyone with a browser.

[Piskvorrr]

You will see it explode indeed (where "it" means "the browser" and "explode" means "tab gets throttled and/or crashes"). I understand the motivations for everything-in-a-browser, but this approach has major externalities - e.g. the client's power, both as in CPU and as in battery.

[Klathmon]

Something like a torrent network is a bunch of IO with some processing which fits perfectly with javascript. WebAssembly might be able to get around some of the performance issues in specific areas (like some of the more complicated hashing stuff) if there are some, but it's more than possible to write performant code in javsacript.

[Piskvorrr]

It might be possible, but I have yet to see it. So far, the performance increase has been thanks to the engine executing such code, despite the opposite effort ;) of the JS code.

[lqdc13]

https://webtorrent.io/

[Klathmon]

sadly I believe that relies on special software bridges or gateways to talk to "true bittorrent networks".

[jorvi]

Vuze already has a supporting plugin, and there's been talk for Transmission to include it. I'm not entirely sure, but I think that if 1 person has a compatible client it acts as a bridge connecting the two types of swarms via DHT!

[boyce]

Yes please, someone do this

[dandelion_lover]

> p2p twitter alternative

here you go https://en.wikipedia.org/wiki/Twister_%28software%29

[RJIb8RBYxzAMX9u]

> They've decided to move forward with SHA256 truncated to 20 bytes [...]

I wonder why not SHA512? It's actually faster to compute than SHA512 on 64-bit architectures.

[silotis]

SHA256 was chosen over SHA512 because the two most popular 64-bit ISAs (x86-64 and ARMv8) both define instruction extensions to provide hardware acceleration of SHA256. Hardware support makes SHA256 much faster than SHA512 in software, even on 64-bit processors.

[RJIb8RBYxzAMX9u]

Huh, I didn't know that's the case for x86. My quick search suggests that support is still quite limited though, and none of my HW (1~4 yrs old) support it. I suppose by the time the spec is finalized and widely adopted HW-accelerated SHA256 would be, too (and I should switch my zfs checksum back from SHA512/256 to SHA256).

[AbacusAvenger]

I haven't found any Intel hardware with the SHA extensions actually implemented. I was expecting it in Skylake but apparently that's not the case... Not sure when we can expect to see them.

[silotis]

Right now the most common x86 CPU which implements the SHA instructions is AMD's Ryzen. The only available Intel CPU which supports them is Goldmont, the latest generation of Atom CPU. It seems likely that Cannon Lake and possibly Coffee Lake will also implement them, but I haven't seen any rumors pointing one way or the other.

[mtgx]

Intel has recently implemented SHA1 in hardware. So give it another decade or so.

[ko27]

SHA-1 & SHA-256 were both implemented at the same time.

https://en.wikipedia.org/wiki/Intel_SHA_extensions

[baby]

hopefully SHA-3 (or better, the Keccak permutation) is implemented in hardware first.

[ComputerGuru]

https://neosmart.net/blog/2017/will-amds-ryzen-finally-bring...

[mtgx]

The word encryption doesn't even seem to be mentioned there. At the very least it would help against traffic shaping, which you know is coming once net neutrality rules are dead.

[the8472]

Bittorrent already had an obfuscation protocol wrapper for years. It was effective for a while but the companies that implement traffic shaping equipment stepped up their game and probably rely on traffic flow matching now.

It is an arms race that is not won by updating a slowly evolving core protocol.