[laumars]

The approach I took to my shell was to pass JSON objects about as the primary data type but fall back to text streams when the data doesn't look like JSON.

The problem with passing objects around though is it can cause issues with the parallel nature of pipes. ie the programs in a chain of pipes can run concurrently since the data is being passed is just a stream. But with objects you need to read the object in its entirety to ensure it is a valid object. This means each process is waiting for the previous process to finish outputting it's object before the next can start processing it. While this is less of an issue with smaller chunks of data in a chain, it would become really noticeable really quickly as your data scales up.

I'm sure will be workarounds for the above problem but my project is still young so there are other bugs I'm more focused on (for me I'm more interested in getting an IDE-level auto-complete implemented so writing one liners in the Shell is as natural as writing your favourite language in your preferred development tool.

[still_grokking]

The problems could be tackled nicely by a clean functional solution like Iteratees[1], maybe with a hight-level API like Reactive Streams[2].

[1] https://github.com/playframework/play-iteratees [2] http://www.reactive-streams.org/

[laumars]

Thank you, I will take a read of them.