One thing I dislike about git is that it really does not support empty folders well. Even though they might make sense lot of time. Either now or for future. There is decent reasons to have empty folders.
Just kidding, I don't see how the overhead of the directory entry is even remotely enough to warrant removal. Most of the magic can be left to efficient DEFLATE compatible blocks and removing entries not in the central directory in the first place (ZIP files can support concatenation of new data so long as you re-write the central directory at the end of the file).