[hamburglar]

Reasonable question, although stated kind of harshly by assuming the worst. Sometimes when you know exactly what you need, it’s reasonable to just build that rather than researching all the possibilities of things that could be adapted to your problem. Fear of reinventing the wheel can be a sort of analysis paralysis where you waste a lot of time looking for an overly-generic solution you will never need. It’s a balance.

[kleineshertz]

It's always a balance. I have been working with teams on both side of the fence and I think I am well aware of the dangers of both: keeping the custom wheel running for years vs fighting the particularities of a third-party tool (up to the point they start dictating architectural decisions). Most of the operations Capillaries is intended to perform are row-based, and stellar Spark map-reduce capabilities were not a big selling point, while tech lock-in price seemed pretty high.

On a more general note (Spark discussion aside), I like working with third-party solutions that can do only one thing, but they do it perfectly. And I am ok supporting in-house-built frameworks that behave the same way and do not pretend to be a world peace solution.