[tannhaeuser]

I guess the projects you mentioned are complementary, in that TAP as I understood it merely specs the output stream/protocol that a test run is expected to produce and isn't limited to testing shell scripts, whereas cram/mdx appears more like a user acceptance test convention specifically for shell script code as component-under-test, much in the spirit of "behavioural" testing a la jBehave.

[neuromanser]

just want to clarify that the program-under-test in cram can be written in anything, as long as it's executable and produces some outputs (stdout, stderr (or any other file descriptor, really), files, exit code). right now i'm using cram to test a program written in F#.

it's true that cram shines in tests for programs written in sh/bash/zsh, where you can output the command lines your PUT consists of (while | instead of) running them and then have cram verify that your program composes expected commands.

git-pimp.zsh is an example of an early approach: it runs its commands through a helper which uses $GIT_PIMP_CHATTY and $GIT_PIMP_DRYRUN to optionally output the given command line and whether to actually perform it, respectively. the helper is a function called "o" ([o-impl]) which makes the code look a little like a bullet list ([o use]).

[0120-output.t] sets git-pimp to skip execution of `git mailz` and `review-files` invocations, and echo any `git format-patch`, `git mantle`, ... invocations (regardless of whether they're dryrun or actually executed).

[0120-output.t] https://github.com/roman-neuhauser/git-pimp/blob/master/t/01... [o-impl] https://github.com/roman-neuhauser/git-pimp/blob/master/s/gi... [o-use] https://github.com/roman-neuhauser/git-pimp/blob/master/s/gi...

`o` was a stepping stone and inspiration to [fake]. this is an external command (as opposed to a shell function), so it cannot stub out shell functions, but it provides greater control over the behavior of mocked command lines. eg. one can mock out `rm -rf --no-preserve-root /` but leave all other uses of `rm` to the actual command, define any combination of exit codes and outputs of particular invocations or prefixes, or provide custom implementations for same. if you ever scripted any destructive, hardware- or network-dependent code (ip(8), fdisk(8), ssh(1)), [fake] can help write sort-of unit tests that do not require expensive setup. as always, mocks and stubs are dangerous, and i'm not claiming this is bullet proof, but it's still very useful.

[fake] https://github.com/roman-neuhauser/fake

[bmn__]

You didn't say anything about the output structure of these testing tools. That's the topic of this discussion, not the tools per se.