[lifeformed]

My issue with git is handling non-text files, which is a common issue with game development. git-lfs is okay but it has some tricky quirks, and you end up with lots of bloat, and you can't merge. I don't really have an answer to how to improve it, but it would be nice if there was some innovation in that area too.

[samuelstros]

Improving on "git not handling non-text files" is a semantic understanding aka parse step in between the file write.

Take a docx, write the file, parse it into entities e.g. paragraph, table, etc. and track changes on those entities instead of the binary blob. You can apply the same logic to files used in game development.

The hard part is making this fast enough. But I am working on this with lix [0].

[0] https://github.com/opral/lix

[jayd16]

What's the plan for large files that can't be merged? Images, executable binaries, encrypted files, that sort of thing?

[samuelstros]

Simple left or right merge. One overwrites the other one.

The appeal or structured file formats like .docx, .json, etc. Images are unstructured and simple "do you want to keep the left or right image" is good enough.

[jayd16]

That doesn't really address the game dev use case then. Artists and designers want to prevent conflicts, not just throw away half the work and redo it.

[samuelstros]

track the source of the asset and it works. take ui design. dont track the svg. track the design file itself

[jayd16]

Ok well what if I draw the foreground and you add something to the background and now my changes visually block your changes? Even if the file is merged, our work is wasted and must be redone. P4 is often popular in industry because artists can lock files and inform others that work is being done in that area.

If you actually want to capture those customers it's a use case that needs to be addressed.

[gregschoeninger]

We're working on this project to help with the non-text file and large file problem: https://github.com/Oxen-AI/Oxen

Started with the machine learning use case for datasets and model weights but seeing a lot of traction in gaming as well.

Always open for feedback and ideas to improve if you want to take it for a spin!

[rectang]

Has there ever been a consideration for the git file format to allow storage of binary blobs uncompressed?

When I was screwing around with the Git file format, tricks I would use to save space like hard-linking or memory-mapping couldn't work, because data is always stored compressed after a header.

A general copy-on-write approach to save checkout space is presumably impossible, but I wonder what other people have traveled down similar paths have concluded.

[jayd16]

Totally agree. After trying to flesh out Unreal's git plugin, it really shows how far from ideal git really is.

Partial checkouts are awkward at best, LFS locks are somehow still buggy and the CLI doesn't support batched updates. Checking the status of a remote branch vs your local (to prevent conflicts) is at best a naive polling.

Better rebase would be a nice to have but there's still so much left to improve for trunk based dev.

[zahlman]

What strategies would you like to use to diff the binaries? Or else how are you going to avoid bloat?

Is it actually okay to try to merge changes to binaries? If two people modify, say, different regions of an image file (even in PNG or another lossless compression format), the sum of the visual changes isn't necessarily equal to the sum of the byte-level changes.

[jayd16]

The best solution I've seen is prevention.

What you can do in P4 is work in trunk, make sure you have latest and lock binary files you're working on. If you do that you won't have conflicts (at the file level anyway). Unlike git's design, this collaborative model is centralized and synchronous but it works.

Git is missing the built in binary support, the locking, and the efficient information sharing of file status tracking. With LFS you can cobble something together but it's not fast or easy.

I'm all for other solutions but I wish git would at least support this flow more whole heartedly until something else is invented.

[lifeformed]

I guess a good way to preview things would go a long way. Merging might be unrealistic to generalize, but being able to diff on images, game engine prefabs, audio, etc would help deciding which one to use.

[miloignis]

I really think something like Xet is a better idea to augment Git than LFS, though it seems to pretty much only be used by HuggingFace for ML model storage, and I think their git plugin was deprecated? Too bad if it ends up only serving the HuggingFace niche.