[late2part]

I evaluated moving 2PB with snowball v. putting in 10g/100g links. The issue with snowball ( i started a company that did what snowball did and shut it down (failed )) and other fedex/RAID solutions is that you have 3 transfers. You think the LAN transfer will be quick - but you're generally rate-limited by the systems more than you are by the bandwidth-delay product. If you're in a high traffic DC area, it's pretty easy to get temp bw or install circuits to carry that. 10g for 2pB is 18 days of transfer - which sounds like a lot - but that's 5 days of transfer on each site, 1 day of setup, and 1 week of shipping. Those numbers aren't real but they're close.

So, snowball works in a lot of area, but like so many AWS products, it works if you adapt to it.

pigz/scp/zstd works extremely fast in line.

In your case you're pulling from S3 to another object store.

I moved ~1PB from one S3 region to another. "Why not use replication," they asked. That only works if it's turned on when you upload the object - another fine-print 'gotcha' in the easy AWS service. Then you get into rate-limits. In 2010 I asked AWS if I could spin up 1000 servers to test something - nope - elastiticy at that level is for the big boys.

Now I work for a large cloud company and we still run into elasticity.

To move the 1PB from one S3 region to another we spun up hundreds of spot instances (oh, we were compressing and glacierizing it too) and built a perl/mysql batch job "s3 get | zstd | s3 put" process and parallelized it. One thing nice about S3 is it pulls the md5 hash - unless multipart, in which case it's the hash of the hash, oh yeah.... So you should split it in advance if you want to verify the hash (more fine print).

Worked great. Good for you for sharing this project, very cool.

[rsync]

"10g for 2pB is 18 days of transfer - which sounds like a lot - but that's 5 days of transfer on each site, 1 day of setup, and 1 week of shipping."

I can confirm this, to some degree.

We have larger customers with 20 or 40 or 80 TB of data to bring into rsync.net and everyone is always very interested in physical delivery, which we offer, but it's always easier to nurse along a 20-30 day transfer than ship JBODs around.

As long as you have a transfer mechanism that can be resumed efficiently (such as zfs send) and you don't have terribly bad bandwidth, we always counsel to just run the very long transfer. It does help that we are inside he.net in one location and two hops away from their core in two other locations and we can just order 10gb circuits on a days notice ... because he.net rocks.

[toomuchtodo]

I’ve never heard a bad thing about Hurricane Electric as a bandwidth/colo provider. Happy to hear their reputation persists.

[semi-extrinsic]

As a physicist I've always found the name "elastic scaling" funny. If it's elastic in the physical sense, it means that the energy required to grow to some size is quadratic (or higher) in the size. The marketing meaning is "easy scaling", but the physical meaning is "really hard scaling".

E.g. compare a soap bubble versus a bubble gum bubble. It's a lot easier to scale up the soap bubble, which is not elastic.

[javajosh]

It's a very good observation, and I think it's more than just a funny aside. The word 'elastic' connotes increasing resistance as the cluster grows, but this is a false intuition. From AWS's POV 'resistance' to adding a node is generally small, fixed, and, in general, independent of cluster size. I suspect this is what makes cloud computers in general, and EC2 in particular, such a cash-cow.

Moreover it turns out that elasticity is a very valuable quality of a cluster for most workloads; we want this intuition to be true, that our cluster meets resistance as it grows, in the sense that it will shrink when the workload decreases. This matches our economic intuition, too. We want this so much we have to build another software layer to make this happen - e.g., k8s.

[cheerlessbog]

"Quantum leap" is similarly misused to mean"big change" when it's physical meaning is smallest physically possible change"

[tedk-42]

We did a Snowball transfer of 150TB (mostly media files) from our on prem DC. Cost is one thing we really failed to plan on. You're charged per day you have the snowball (in our case 3 for 2 separate DCs).

During the transfer, the AWS sync constantly failed due to random issues which drove up the total time to transfer the files. Something like having a tilde (~) in the filename will totally break the sync. You really need to keep track of where it failed. We were constantly trying to craft additional rules into our sync logic to catch the 'gotchas'.

Another point you alluded to was the Etag/MD5sum that's stored in AWS. Pretty useful if you know how to use it...