[JamieLewis]

Awesome idea.

This week as been all about getting a second version stamped and out (nowhere near a finished version or even 1.0, but a 0.2 version)

I've been working on https://ghostream.com a framework for constructing stream processing systems. Getting a first draft of the website (http://imgur.com/ujsmjJz) and the documentation (although there is still much to do)

And after that, lots of bug fixes and environment clean ups (code coverage pipeline / static analysis etc.) Managed to knock down quite a few little bugs that way.

[mtrn]

Interesting. I'm still searching for a powerful, yet simple to use stream processing library. Which projects inspired you to write ghostream.com?

[JamieLewis]

I've played with a variety of frameworks (Storm and IBM Infosphere Streams are probably the two I have worked with the most) - In my previous job I was in a team which designed and developed our own (and later integrated partly with IBM Streams) I took down many lessons from the initial creation, the later integration and the work following that. ghostream was born out of those lessons.

Mostly I found the follow problems with existing frameworks :

- Tied to an execution environment - Storm with ZooKeeper and Streams with their own custom one. This makes it very difficult to use either for small projects - and limits integration choices when scaling.

- Resource hogs - The IBM Streams environment is a huge resource hog, I never really got that far with Storm but the number of dependencies it required just to get something up and running provided a similar sense of dread.

- Easy to debug/optimization - Streams has the best tools for this at the moment but they are all heavily tied to the execution environment. I'm hoping with ghostream that structures can be built at the protocol level to provide a way to build tools on top - not tied to any particular environment.