[thesmallestcat]

Great advice! I'd venture that managing large files is an unsolved problem. It's a hack in most version control systems, and uploading/downloading files from a host, even S3, is a slow, serial process. Same for checksumming. Network speeds have more than caught up, and large files are a frequent process bottleneck. Something that makes it easy to manage and consume large files could be a big deal. It probably would require a new application protocol, maybe even a new filesystem similar to XFS.

[imhoguy]

Or maybe a file should stay where it is and processing logic itself should be deployed there. If file parts are distributed then processing could be suspended and migrated to place where next piece is stored. Something similar is done with Hadoop and HDFS.