[brian_cloutier]

This is technical recruiting done right, a lucid walkthrough of a hard problem complete with links to the implementation. It must have taken at least a few weeks of engineer-time to write, Github is awesome for making this public.

I wish they had talked a little more about the tradeoff they made. They mentioned that splitting packfiles by fork was space-prohibitive, but ended up with a solution which must take more space than originally used. (If the new heuristic refuses to use some objects as delta bases, some options which would have provided the best compression are no longer available to git)

The performance win is incredible, how much space did they give up in the process?

[peff]

The fork-aware packfiles are bigger, but not as much as you might think. We give up the "best" delta that git can find, but usually there is another one in the same group that is almost as good.

I didn't have any numbers on hand, so I just repacked our git/git network.git with and without the delta-aware code. A stock repack is about 300MB, and the delta-aware one is 400MB.

That sounds like a lot, but the real alternative is not sharing objects at all, which is more like 350GB.

[omouse]

This is an actual interesting problem that takes skill and knowledge to solve and it actually produces immense business value too (at least for Github). This is the kind of stuff more software engineers should be working on. I'm dying over here working on a small React JS component and some CSS layout tinkering, my brain's got graph traversal algorithms and bitmap indexes on its mind and can't do anything about it.

[mikekchar]

Every large problem is a series of small problems. I've worked on large complex systems and on small toy problems. It really isn't much different except for one thing. With toy problems you can get away with making a lot of mistakes because the interactions are simple. As the system becomes more and more complex, your execution as a programmer becomes more and more important. You have to think very deeply about the "why" and you have to be able to test your assumptions well.

If you have ever played the game of go, when you first start out the board is empty. You have to place your stone somewhere, but actually it doesn't really matter where. Over time, as you place more and more stones, your choices become more and more important (and ironically, you have more and more constraints). The weird thing is that, even though it doesn't matter where you place those first stones, it becomes very important where the stones were originally placed as the situation unfolds.

Programming is similar. When you first start a project, it really doesn't matter what you do. Almost everything will work to one degree or another. But the original decisions gain more and more weight as the project becomes more and more complex (and you are faced with more and more compromises). Eventually those original decisions can make or break your project, even though it didn't matter at first what they were (of course, this is why refactoring is so important --- but that's a different discussion).

My point (finally) is, that even though you may be making simple changes on simple systems, it doesn't have to stop you from understanding the implications of your work should the project become larger. Take the opportunity to polish your skills and make your "opening game" as perfect as you can make it.

I agree that at some point every programmer must start working on complex systems in order to grow. If you are at that point and your employer does not offer complex problems, then maybe it is time to move. However, don't neglect your "opening game". It is very, very important because, as I said at the beginning, every large problem is a series of small problems.

[omouse]

I only recently started playing Go and the similarity that you point is fantastic.

Take the opportunity to polish your skills and make your "opening game" as perfect as you can make it

Yep, exactly what I've been doing.

If you are at that point and your employer does not offer complex problems, then maybe it is time to move.

Yep ;-)