[lifeisstillgood]

Doesn’t wireshark have a format for storing request / responses ? That seems a fair standard to lean on.

I like the idea - I think I have at least two formats for storing expected request / responses and probably more.

But standardising - as in not just my ball of twine tools uses but every uses is great.

I just think it already exists ?

[codesoap]

To be frank, I hadn't even considered looking at the file format of wireshark when thinking of existing file formats, that I could reuse. I've now taken a brief look and it seems like wireshark supports quite a lot of different formats [1], but the preferred one seems to be PcapNG. At a first glance, there are several attributes that make them less suitable for my purposes:

1. PcapNG as well as the other file formats look like they are storing packets, which is a lower level than HTTP requests and unnecessarily verbose for my intended purposes.

2. They are binary formats, which makes them less suitable for printing to stdout. This also means, that they are not line based, which means UNIX tools, like grep, cannot be used effectively.

3. They are not designed for streaming. The httpipe format is line-based and contains no header/global fields. Thus it is trivial to, for example, build a filtering program: it would just read one line at a time and print it again, if it matches filter criteria; the output would automatically be valid httpipe again.

4. Lastly, parsing and composing JSON is something most developers have done before and basically every programming language has libraries for it. This makes it easy for the ecosystem to grow and enables users to build custom tools without too much initial effort.

[1] https://wiki.wireshark.org/FileFormatReference

[2] https://pcapng.com/

[lifeisstillgood]

Fair enough. This does seem like one of those things that’s so simple that no-one thinks “oh I will grab the library to work with it”. That’s kind of a good thing, I mean JSON is so simple but I still use a library for reading and writing

[junon]

I implemented my own packet capture stuff for Wireshark, the format is pretty straightforward. Use case was different and I reimplemented the rpcapd protocol but the packets themselves are easy to dump assuming you have access to the raw packet information (Ethernet headers and whatnot). You can of course also synthesize that information if needed.