[jack9]

Kafka was useful in a project because of the semantics for what amounts to writing to file from network sockets.

Spark is a terribly inefficient solution to any known/stable data processing or analytics jobs. If you want a common format to trade with buddies, it's useful now. I expect something else will come along to replace that fad tech.

[tabacof]

>Spark is a terribly inefficient solution to any known/stable data processing or analytics jobs.

Can you expand on that?

[mdekkers]

Spark is a terribly inefficient solution to any known/stable data processing or analytics jobs.

You gave a typo there.

Spark is a terribly inefficient solution to any known/stable data processing or analytics jobs I have ever come across.

There, fixed that for you.

[jack9]

I don't think that's fixed, because it's not what I assert.

Any job that is predictable, can be done faster and cheaper with some C (or Go) and ad-hoc delegation to load balanced VMs. Spark is difficult to optimize for consistent processes, poor on resource usage, and requires specialized knowledge. Just pay for someone who has done a little embedded programming and stop creating buzzword jobs because a prototype went to production without comparison.