[jermaustin1]

Do you chunk the upload so if I lose internet for a second, it can just upload the chunks that failed?

This is an issue I have with one of my employees. They only get consumer Comcast, and the connection is crap. I ended up writing a bash script that will chunk and scp the files to a central server, and if it fails a chunk, it leaves it in the directory for her to manually re-run.

That said, I pay $20/mo for the file server (which i would no matter what) and it has 3TB of outbound bandwidth and unlimited inbound, and is less than 5% of your cost.

[davehorne]

Although we do chunk the upload and send the chunks in parallel to the server we do not at this point have the recover functionality implemented. We are able to recover from a disruption in internet connectivity with retry logic in javascript but if your computer crashed completely we would not recover. That is on the roadmap and expected to be released in the next couple of months though.

[davehorne]

In terms of cost there are certainly cheaper ways to accomplish transfers. MASV is designed to be very fast and we have 9 servers across the world to enable transfers anywhere. If your doing local transfers your own file server is likely good enough but if your sending hundreds of GB's cross country or across the world you will likely run into major performance issues. It really comes down to how much your time is worth and if file transfer are a consistent requirement for your business. This is certainly intended to be a B2b tool less for consumer use.

[microcolonel]

> I ended up writing a bash script that will chunk and scp the files to a central server, and if it fails a chunk, it leaves it in the directory for her to manually re-run.

Use Syncthing, it traverses NATs, it is trivial to set up, it is seemingly very secure by its nature, and crucially for you, it transfers in blocks. You can administer it through a web interface with credentials (or with none, if it's just listening on loopback).

[davehorne]

A lot of why our users enjoy MASV is because it's dead simple to use for non-technical users. Not that syncthing is very hard but as a video pro dealing with clients the easier it is for them to just click a download button the more likely it will happen. Also because it's pay-as-you-go it's easy to charge back the transfer cost as a part of the project bill.

[microcolonel]

Most certainly. Another major benefit is that you folks provide storage, which inherently simplifies administering something like this. One benefit of Syncthing/BT Sync that I have not seen emulated by other vendors is peer-to-peer transfers, which can be considerably faster if both peers are on the same network or an adjacent one (including one on the same provider, or connected to the same exchange, or if the central service has no nodes on the same continent as the peer). The performance is just unmatched for such transfers, in my experience. A peer-based protocol could be a considerable help even if you don't intend on allowing direct peer-to-peer transfers, because (if implemented with sufficient care) it can allow you to choose better nodes to maximize throughput and integrity.

I think there is a very legitimate need for centralized administration, I have all of my Syncthing node configurations synchronized over Syncthing itself (which is a bit of a risk, but not overly so), but I would consider paying for something (if I could have similar confidence and flexibility with the client software) more integrated, with some safeguards against accidentally disabling configuration sync (which would require me to directly configure the node) to a specific node, and some integrated ability to have nodes self-report and self-allocate.

[davehorne]

Absolutely agree. We really built our network out to be independent of our file transfer tool. It creates routes across any cloud provider and can integrate into any cloud storage provider. It also uses machine learning to determine which routes are optimal based on previous performance and time of day. I see a future where you could use our network for many different products such as an accelerated VPN, streaming live content, or as you pointed out as a pass through network for a P2P connection if both endpoints are available but it would route across our network to avoid congestion. Potentially even being able to make a copy of the transfer in the cloud for archive. Even P2P relies on routing across the public internet which is prone to congestion and is set up to do least-cost routing for the ISP's. Usually it's less of an issue though because the protocols help ensure it pushes through that congestion. Lot's of cool possibilities for this in the future and nowhere near enough time!

[jermaustin1]

I'll have to check that out. I was just desperate one afternoon after dropbox was failing to sync because of connectivity issues, and I knew bash just well enough to split, scp, rm, repeat.

[rocmcd]

You may also want to look into using rsync as an alternative to scp. rsync is much smarter about how it syncs data, so it should be better for your use case.

[davrosthedalek]

Look at rsync. rsync -avP file host:/path/on/host Repeat if it fails. It will not retransfer already transferred data.

[Retric]

Is your 20$/mo server 1Gbit? At a fairly common 100Mbit these file transfers would be 1/10 the speed and also need to be downloaded. Even slower if you need to send to multiple people.

Running bit torrent let's peers share the data you upload and get the same net effect as this. But, that's not as far as I know a browser plugin.

PS: I don't think it's a big deal, but executed well it might be a valuable niche.

[jermaustin1]

It averages 400Mbit upload and 350Mbit download, not that it matters to me personally, as my employee can rarely upload faster than 4Mbit.

That said, their 1Gbit connection will probably average around the same speed if they have more than a few users uploading 50GB files during normal business hours.

[davehorne]

Just to clarify we will fill a 1 Gbps pipe but our servers are provisioned to do much more than 1 Gbps. If many users are uploading at the same time we scale up beyond 1 Gbps that's not our network bandwidth maximum.

[jermaustin1]

That's good to know. 1Gbit is our "max" but that works just fine for the company's "network share" If we ever were to start hitting our bandwidth caps, I can pay to increase those, but the speed is pretty much fixed without changing providers. If speed were to ever be an issue, I would probably look at a different service like yours. It would be a massive hit at your current pricing, though.

[justinsaccount]

huh? Why didn't you just use rsync?

[martin-adams]

I presume you know that rsync is a better solution to the problem. Not all of us do. Would you mind explaining why rsync is a better solution?

[vortico]

For starters, it's a proper replacement for "a bash script that will chunk and scp the files to a central server, and if it fails a chunk, it leaves it in the directory for her to manually re-run". It's installed on almost all Linux and Mac systems, so if your customer is using a script already, they might as well use rsync.

[seszett]

> Would you mind explaining why rsync is a better solution?

rsync basically chunks the files and sends the missing (or differing) chunks over scp, leaves the files in the destination directory if something fails, it is more or less an automated tool that already does what the GP is talking about.

[mbesto]

> it is more or less an automated tool that already does what the GP is talking about.

But requires someone to be comfortable using a CLI.

[j1elo]

But that 'someone' is someone who is already running a Bash script and re-running it each time an error occurs. So I don't see the problem here about using a CLI.

[davehorne]

For technically capable people there are certainly lot's of good options for sending lots of data fast. We are just trying to make a service that makes this more accessible and requires no setup. As a side note we are currently working on an API and we intend to integrate with rsync so it can be used to upload content to our network.

[mbesto]

I read it as "I'm running a bash script for someone else so they can pick it up from a server", not "my employee who has this issue created a bash script"...maybe I misinterpreted.

[teilo]

Because we have to deal with people as they are, and not as how we want them to be.

This is for sending large assets from party A to party B, where either the parties do not have access to a shared rsync-capable server, or they do not know how to use rsync. On a deadline, you can't afford to become a CLI trainer or require special software.

A common use case would involve service bureaus and creators. Trust me when I say that there are a hell of a lot of creators out there who would never be able to manage rsync. You want to make stuff like this drop-dead simple for people. That means a service that works in-browser.

[davehorne]

Couldn't have summarized it better myself. I like to think we are just making cloud services more accessible for non-technical people and adding a tax to that. There is always ways to get something cheaper by rigging up a setup yourself but for creators that is time that could be spent on creating instead of infrastructure or tooling.

[justinsaccount]

I was asking the person who said

> bash script that will chunk and scp the files to a central server, and if it fails a chunk, it leaves it in the directory for her to manually re-run

not responding to the linked post itself.

[jermaustin1]

Mainly because I knew how to use split a file and use scp already. I didn't feel like researching the options and syntax of a new CLI tool.

[halbritt]

I use a javascript library that works in the browser and supports S3 file copy chunking and resuming. Host that in a simple S3 bucket, create a target bucket with acceleration enabled and bob's your uncle.

It's very efficient. I've never seen it fail to consume all of the available upstream bandwidth from the client. If this is of interest, I can go dig up the library I used.