[mschwaig]

I've been thinking if this could be done in Nix and toyed around with the idea just a little bit using nano-gpt.

Hermeticity always seems to mean isolation, but depending on who you ask it does not always mean computing some sort of hash over all build input as the 'identity' of a particular step in the pipeline, like Nix does.

If you do that hash-based identity part, looking up intermediary results and resuming from there happens using this sort of hash.

Does savanah do that, or will it resume where it left off based on a less strict notion of identity?

I could see arguments for either approach.

[michaelchicory]

Savanna works in the same way as Nix: each training stage is defined fully in code, so it computes a hash over all inputs (training image, code version, environment, training configuration, etc.) and maps this to the eventual output. When the same inputs come along again, Savanna just reads the output from cache. As the post mentions, this is particularly helpful in sweeps that span multiple training stages.