[willbeddow]

Juice is cool, but tradeoffs around which metadata store you choose end up being very important. It also writes files in it's own uninterpretable format to object storage, so if you lose the metadata store, you lose your data.

When we tried it at Krea we ended up moving on because we couldn't get sufficient performance to train on, and having to choose which datacenter to deploy our metadata store on essentially forced us to only use it one location at a time.

[tptacek]

I'm betting this is on the front page today (as opposed to any other day; Juice is very neat and doesn't need us to hype it) because of our Sprites post, which goes into some detail about how we use Juice (for the time being; I'm not sure if we'll keep it this way).

The TL;DR relevant to your comment is: we tore out a lot of the metadata stuff, and our metadata storage is SQLite + Litestream.io, which gives us fast local read/write, enough systemwide atomicity (all atomicity in our setting runs asymptotically against "someone could just cut the power at any moment"), and preserves "durably stored to object storage".

[ghm2199]

Litestream.io is amazing. Using sqlite as a DB in a typical relational data model where objects are related mean most read then write transactions would have to one node, but if the using it for blobs as first class objects(e.g. video uploads or sensor data) which are independent probably means you can shard and scale your set up the wazoo right?

[c4pt0r]

In large-scale metadata scenarios, JFS recommends using a distributed key-value store to host metadata, such as TiKV or FoundationDB. Based on my experience with large JFS users, most of them choose TiKV.

Disclaimer: I'm the co-founder of TiKV.

[suavesu]

Truth. It supports very large volume, more than 100PiB and 10B files in a single volume.

Disclosure: I'm co-founder of JuiceFS

[AdamJacobMuller]

> It also writes files in it's own uninterpretable format to object storage, so if you lose the metadata store, you lose your data.

That's so confusing to me I had to read it five times. Are you saying you lose the metadata, or that the underlying data is actually mangled or gone, or merely that you lose the metadata?

One of the greatest features of something like this to me would be the ability to durable even beyond JuiceFS access to my data in a bad situation. Even if JuiceFS totally messes up, my data is still in S3 (and with versioning etc even if juicefs mangles or deletes my data, still). So odd to design this kind of software and lose this property.

[mrkurt]

It backs its metadata up to S3. You do need metadata to map inodes / slices / chunks to s3 objects, though.

Tigris has a one-to-one FUSE that does what you want: https://github.com/tigrisdata/tigrisfs

[ifoxhz]

FUSE generally has low overall performance because of an additional data transfer process between the kernel space and user space, which is less than ideal for AI training.

[suavesu]

The MLPerf performance of JuiceFS last year, FUSE client, and TCP/IP network. There're some analysis about performance and bottleneck. -> https://juicefs.com/en/blog/engineering/mlperf-storage-v2-ai...

[cbarrick]

As I understand it, if the metadata is lost then the whole filesystem is lost.

I think this is a common failure mode in filesystems. For example, in ZFS, if you store your metadata on a separate device and that device is destroyed, the whole pool is useless.

[suavesu]

metadata backup is very important. don't forget.