Hacker News new | ask | show | jobs
by pjc50 927 days ago
Some years ago (1996) I wrote a GIF decoder from scratch based entirely on http://qzx.com/pc-gpe/gif.txt (and a description of LZW bundled with my copy of pcgpe that seems to be missing from that page) . I remember it being quite a struggle against off-by-one errors.

I do wish there was a really good format for describing binary file formats in a way that was amenable to codegen. Kaitai https://kaitai.io/ seems to be the state of the art.
1 comments
yjftsjthsd-h 927 days ago
Kaitai looks nice - have you used it enough to review how it handles? I'm just starting a project to deal with somewhat involved on-disk formats[0] and this might be helpful.

[0] The other day, someone was asking for a "tar2ext4" tool, and I thought "hey, that should exist, and I need a side project!". I was prepared to use an annotated hex viewer ( https://hachoir.readthedocs.io/en/latest/wx.html ) and hand roll the encoder/decoder, but I'll happily take tool assistance:)

link
pjc50 926 days ago
Kaitai is good if:

- your format is fully known (it's less helpful if you're trying to incrementally build a parser while reverse engineering)

- you want to read files, but don't care about writing

- you don't mind that the development is not very active

For writing "tar2ext4" I would genuinely look at how much work it would be to run the ext4 code from the kernel in a different context; there's a lot of it to consider. Or do what the Apple "dmg" tooling does and make a ramdisk.

link
yjftsjthsd-h 926 days ago
Okay, so of interest but maybe not applicable to my usecase. Thanks:)

Yeah, it remains to be seen how complex the actual format/code is. Would need to balance the difficulty of recreating it (which I assume to be quite high) against difficulty of extracting kernel code... although https://github.com/lkl/linux exists so for all I know maybe it's easy¯\_(ツ)_/¯

And yes, if I needed to actually write a "tar2ext4" tool today - like, start working in the morning and have it done by EOD - I would absolutely use... actually probably a loopback device rather than a true ramdisk, but yeah. But that requires root access and fiddling with loopback config, which seems excessive for what is, ultimately, just another archive format (from a certain point of view). And honestly some of it is just that it sounds fun to get my hands dirty with filesystem code in userspace:)

link
yjftsjthsd-h 923 days ago
(Forgive my replying to myself; I'm out of the edit window and this seemed good to leave in case someone stumbles across this comment)

Yeah so I'm maybe a touch dumb and missed this right in the lkl readme - their demos all but include that exact function:

    fs2tar - a tool that converts a filesystem image to a tar archive

    cptofs/cpfromfs - a tool that copies files to/from a filesystem image

    lklfuse - a tool that can mount a filesystem image in userspace, without root priviledges, using FUSE
link