|
|
|
|
|
|
by jamesblonde
2814 days ago
|
|
|
We think that there is a niche for a higher performance dist FS than S3. We have integrated NVMe hardware with our HDFS implementation (HopsFS) and made its metadata layer distributed. NVMe means you can, for example, work with datasets with millions of files for deep learning - instead of having to munge them into parquet files because your FS is slowing down your machine learning pipeline. Reference:
https://www.logicalclocks.com/millions-and-millions-of-files... We have also redesigned the stack around our distributed metadata layer. We are primarily targeting on-prem right now, but HopsFS would be the fastest DFS in the cloud if you ran it there today. |
|
|