[gijzelaerr]

Interesting. I've been working on something related, a non intrusive docker 'extension', kliko. Kliko is a specification to formalize file based (no network yet) input/output flow for containers. It makes to possible to have a generic API for containers, so you can automatically create user interfaces for an application or chain them together in a pipeline.

https://github.com/gijzelaerr/kliko

Here is an example kliko file for a container defining the input and output:

https://github.com/kernsuite-docker/lwimager/blob/master/kli...

[valarauca1]

So one process writes to a flat file, then another process reads the flat file? Wouldn't named pipes be more useful here?

Also it feels like you are just making a seralization format on top of Stdin/Stdout.

[gijzelaerr]

There is doc available on the website, but I'm not sure if I explain the concept properly.

The assumption is that you have containers that operate on input files and generates output files. The behavior of the container depends on the given parameters which are defined in the kliko file. The user (or runner) will supply these parameters at runtime.

To illustrate, we use it for creating pipelines in radio astronomy where we operate on datasets of gigabyes or bigger. most of these tools are file based, they read files in and write files out. It is all quite complex and old software, so Docker is ideal for encapsulating this complexity. A scientist can easily recombine the various containers and play with the arguments. By the split of input/output the container effectively become 'functional', no side effects and the results can be cached if the parameters are the same. The intermediate temporary volumes can be memory based to speed things up. We use stdout for logging.

[ymt123]

If I'm understanding correctly that sounds similar to Pachyderm (http://www.pachyderm.io)

[IanCal]

Pachyderm looks quite cool, but I think it's lacking a quick-start and a way of running things locally in a simple way.

I grabbed the repo and clicked through a few links in the docs and hit a 404. I searched on google and found a link to a way of running it just locally simply but that doesn't work with the new version. Then I followed the instructions and hit a problem installing something to do with k8 about mapped paths and the fix printed in the console doesn't work.

I understand that this is a personal complaint and others might not care at all about having it setup locally because it solves the big problems so well but I just want to try it at least locally.

[bane]

This would likely benefit other scientific fields -- bioinformatics for example has the same sort of software tooling.