[lobster_johnson]

Cool, thanks! I haven't looked closely at it, but the "version controlled" part is something I don't need/want. Does it get in the way at all? I'm mostly looking for incremental, semi-real-time streaming processing, not something where you shoot off a big job on a dataset and get back results.

[jdoliner]

The version control semantics of the system are pretty crucial for some of the features you describe wanting. Pachyderm supports incremental operations on stream-like datasets. But what's going on under the hood is that the dataset is being version controlled and thus the system can tell which data has changed and only process the new data. Hope that helps, I'd be happy to chat more about your specific use case. Shoot me an email at [e-mail]