[celoyd]

Heh, yes, we seriously considered the Honda Civic full of external drives option.[0]

Some of our images were from NASA’s hot new GIBS[1] service, which was extremely fast. However, it’d only backfilled about 2/5 of the data we needed, so for the rest of it we were using a legacy endpoint that was launched around 2004ish and has some, let’s say, idiosyncratic caching and throttling systems.

Instead of setting up a special channel, we talked to them and figured out how to shotgun the downloads in a way that wouldn’t kill their cache or require them to give us special treatment. We thought of this on a Friday and wanted to have it ready when we came in on Monday, and it worked. Thus the somewhat blunt methods.

0. For any young persons in the audience, the reference is to “Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.” from Tanenbaum’s “Computer Networks”.

1. https://earthdata.nasa.gov/about-eosdis/system-description/g...