Hacker News new | ask | show | jobs
by makmanalp 1784 days ago
FWIW the scientific computing community (who often deal with petabytes of geodata) has been thinking of ideas like this for a while, e.g. techniques around file formats that are easy to lazily and partially parse, (ab)using FUSE to do partial reads using http RANGE requests, some combination thereof, etc:

http://matthewrocklin.com/blog/work/2018/02/06/hdf-in-the-cl...