Hacker News new | ask | show | jobs
by thescriptkiddie 3869 days ago
Resumable file transfers in JS is really cool.

But why a totally new protocol? Why not use bittorrent, rsync, or even ftp? A lot of effort has gone into making those reliable in the face of network congestion. Trying to shove a lot of bits down a single TCP connection with no traffic shaping is just asking for bufferbloat to strike.

3 comments

Because these protocols are unusable from the perspective of JS on a webpage in a browser. Yes, browsers do support FTP, but I don't believe there are any browsers that support FTP uploads (and even if there are any, I doubt it'd be possible to upload from JS).

HTTP also has the advantage of being allowed through virtually every firewall.

It seems that webapps are paving the way of the future, for better or worse.

Hi, one of the authors here. Thanks for raising a valid concern. Vimeo in fact already offers FTP uploads for premium users, but helped us write this as the advantages of something HTTP based are significant. We can:

- be sure that browsers and servers already speak it (so small additional libraries required on both ends)

- gracefully degrade to regular form POSTs should tus support not be possible (IE8, etc)

- easily write hooks for progress bars, and e.g. encoding of the uploaded material

- use existing holes in firewalls. FTP requires many ports (for PORT, DATA, PASV) that are blocked on airports, public libraries, large corporations

- add existing HTTP components to make tus better (loadbalancing, intrusion detection, auth, proxies, etc). Some of these apply to FTP as well, but the HTTP options are more (advanced).

Bittorrent is not really client-server oriented, which is what this protocol is trying to solve. Rsync has many the same disadvantages as FTP listed above.

As for a single TCP connection with no traffic shaping, tus support splitting up a file into multiple parts and uploading them in parallel (as regular tus uploads, so profiting from checksums, retries, and resumability) and stitching them together on the server-side by issuing a 'concat'. This means you can have as many connections as you like, and the individual chunks can complete in any order as well.

Traffic shaping is not part of the protocol. By default tus should saturate available connections.

That said we're writing down some recommendations in a separate 'developer guide' document that offers best practices for implementing and deploying tus, traffic shaping could be part of it. Feel free to weigh in here https://github.com/tus/tus-resumable-upload-protocol/pull/68

rsync was the first thing that came to my mind as well. You can even use it for entire directories of stuff.