[microcolonel]

> I ended up writing a bash script that will chunk and scp the files to a central server, and if it fails a chunk, it leaves it in the directory for her to manually re-run.

Use Syncthing, it traverses NATs, it is trivial to set up, it is seemingly very secure by its nature, and crucially for you, it transfers in blocks. You can administer it through a web interface with credentials (or with none, if it's just listening on loopback).

[davehorne]

A lot of why our users enjoy MASV is because it's dead simple to use for non-technical users. Not that syncthing is very hard but as a video pro dealing with clients the easier it is for them to just click a download button the more likely it will happen. Also because it's pay-as-you-go it's easy to charge back the transfer cost as a part of the project bill.

[microcolonel]

Most certainly. Another major benefit is that you folks provide storage, which inherently simplifies administering something like this. One benefit of Syncthing/BT Sync that I have not seen emulated by other vendors is peer-to-peer transfers, which can be considerably faster if both peers are on the same network or an adjacent one (including one on the same provider, or connected to the same exchange, or if the central service has no nodes on the same continent as the peer). The performance is just unmatched for such transfers, in my experience. A peer-based protocol could be a considerable help even if you don't intend on allowing direct peer-to-peer transfers, because (if implemented with sufficient care) it can allow you to choose better nodes to maximize throughput and integrity.

I think there is a very legitimate need for centralized administration, I have all of my Syncthing node configurations synchronized over Syncthing itself (which is a bit of a risk, but not overly so), but I would consider paying for something (if I could have similar confidence and flexibility with the client software) more integrated, with some safeguards against accidentally disabling configuration sync (which would require me to directly configure the node) to a specific node, and some integrated ability to have nodes self-report and self-allocate.

[davehorne]

Absolutely agree. We really built our network out to be independent of our file transfer tool. It creates routes across any cloud provider and can integrate into any cloud storage provider. It also uses machine learning to determine which routes are optimal based on previous performance and time of day. I see a future where you could use our network for many different products such as an accelerated VPN, streaming live content, or as you pointed out as a pass through network for a P2P connection if both endpoints are available but it would route across our network to avoid congestion. Potentially even being able to make a copy of the transfer in the cloud for archive. Even P2P relies on routing across the public internet which is prone to congestion and is set up to do least-cost routing for the ISP's. Usually it's less of an issue though because the protocols help ensure it pushes through that congestion. Lot's of cool possibilities for this in the future and nowhere near enough time!

[jermaustin1]

I'll have to check that out. I was just desperate one afternoon after dropbox was failing to sync because of connectivity issues, and I knew bash just well enough to split, scp, rm, repeat.

[rocmcd]

You may also want to look into using rsync as an alternative to scp. rsync is much smarter about how it syncs data, so it should be better for your use case.

[davrosthedalek]

Look at rsync. rsync -avP file host:/path/on/host Repeat if it fails. It will not retransfer already transferred data.