Hacker News new | ask | show | jobs
by Sharlin 33 days ago
> The really neat thing about GGUF is that it's just one file. Compare this to a typical safetensors repo on huggingface, where there's a pile of necessary JSON files scattered around [...]

Funny, to me AI models have "always" been single files, as that's what has been the norm in the local image gen business. Safetensors files allow stuffing all kinds of stuff inside them too, no GGUF needed for that. Though given that the text encoders of modern models are multi-gigabyte language models themselves, nobody includes redundant copies of those in every checkpoint.
2 comments
Philpax 33 days ago
Single-file deployments were an intentional design goal on my part. While most image models were/are single-file, LLM safetensors (at least at the time) were not, and I wanted to ensure that we enforced that at a structural level. I also didn't want to mandate a JSON reader for executors (e.g. llama.cpp), which the ST approach would have required. The bigger issue at the time, if I recall, was that ST couldn't support the new-and-upcoming quants that GGML had, and having our own file format offered us flexibility that ST couldn't.
link
embedding-shape 33 days ago
> to me AI models have "always" been single files, as that's what has been the norm in the local image gen business

That doesn't even make sense in the "local image gen business", you don't use a single weights file, you need a bunch of encoders/decoders and what not to actually be able to run the architecture with the weights.

Maybe the tooling you use hides those things from you, but they're still there under the surface.

link