[embedding-shape]

> Broken files — 16 models were actually Git LFS text pointers (several hundred bytes), not binaries. If you try to load them, your code crashes.

Yeah, if you don't know how use the repositories, they might look broken :) Pointers are fine, the blobs are downloaded after you fetch the git repository itself, then it's perfectly loadable. Seems like a really basic thing to misunderstand, given the context.

Please, understand how things typically work in the ecosystem before claiming something is broken.

That whatever LLM you used couldn't import some specific libraries also doesn't mean the repository itself has issues.

I think you need to go back to the drawing board here, fully understand how things work, before you set out to analyze what's "broken".

[arseniibr]

In an ideal local environment with a properly configured git client, sure. But in real-world CI/CD pipelines, people can use wget, curl, or custom caching layers that often pull the raw pointer file instead of the LFS blob. When that hits torch.load() in production, the service crashes. The tool was designed to catch this integrity mismatch before deployment.

[embedding-shape]

Right, but if your CI/CD pipeline is fetching repositories that are using Git LFS while whatever pipeline you're creating/maintaining can't actually handle Git LFS, wouldn't you say that it's the pipeline that would have to be fixed?

Trying to patch your CI builds by adding a tool that scans for licenses, "malware" and other metadata errors on top of all of this feels very much like "the wrong solution", fix the issue at the root instead, the pipeline doing the wrong things.

[arseniibr]

I agree that fixing the pipeline is indeed the correct decision, but I've created this tool to provide the detection.

In a complex environment, you often don't control the upstream ingestion methods used by every team. They might use git lfs, wget, huggingface-cli, or custom caching layers.

Relying solely on the hope that every downstream consumer correctly handles Git LFS is dangerous. This tool acts as a detector to catch those inevitable human or tooling errors before they crash the production.

[embedding-shape]

> This tool acts as a detector to catch those inevitable human or tooling errors before they crash the production.

Again, that sounds like a bigger issue, that a repository using Git LFS can somehow "crash the production", that's where I'd add resilience first. But as mentioned in another comment, I don't have the full view of your infrastructure, maybe it has to work like that for whatever reason, so YMMV.

[wbshaw]

Calling them broken files might not be correct. However, I can see where if you are not diligent about watching commits to those git repos, you end up with a Trojan Horse that introduces a vulnerability after you've vetted the model.

[embedding-shape]

Well, sure, but how does this tool help in any way with that? Since if you're using Git LFS, the tool just says it's broken, rather than actually pulling down the blobs and checking those. It wouldn't prevent "malicious weights".

Besides, pickle is the data format that introduces the possibility for vulnerabilities, if the model weights are in .safetensor you're safe regardless.