[mrdoops]

A lot of comments not sure about what Graph DBs are good for:

* Flexible knowledge association i.e. Knowledge Graphing

* Modeling and querying associations / models with many-steps-removed requirements

* Expert Systems / Inference Engines

* Lazy traversal for complex job scheduling

Graph DBs are not good at being a general purpose 95% of use cases database. Just use Postgres/MySQL if you're not sure. We use Neptune (AWS managed GraphDB) to model cybersecurity dependencies between many companies and report on supply chain vulnerabilities many steps removed. Those kinds of queries are non-trivial and expensive on anything but a Graph Database.

As GraphDBs meet niche query requirements you usually have other databases involved in the full application. If you want to tractably manage many databases in a system you ideally want to be in streaming / event sourced semantics. If you're already in an imperative crud-around-data / batch pipeline you'll find greater maintenance costs in adopting a GraphDB or any additional DB for that matter.

[jshen]

I have yet to see streaming/event sourcing work well. Every time I’ve seen it used it’s caused more problems than it’s resolved. The main problems, out of order events and/or slow to propagate events.

[mrdoops]

From a technical perspective there are plenty of ways to serialize / order some stream of events in a reasonable way. Whether that's implementing your event store on top of a transactionally secure database (e.g. Postgres, not Mongo) or using a higher throughput, less persistently secure solution like Kafka. There's also ways to deal with eventual consistency and distributed transactions (SAGAs) depending on the need.

The hard part is that ES/Streaming systems work best / almost necessitate a clean and clear domain model. A clean and clear domain model requires a lot of discussion and consensus with domain experts and product owners. Buy in to have the kind of discussions needed is the source of the issues I've experienced with these kind of systems. CRUD can paint over a lot of cloudy abstract concepts for better or for worse. These kind of discussions are energy intensive and mentally painful to cast light on the cloudy thoughts.

There's not great streaming/es support on a language level outside of the robust actor model systems (e.g. Erlang/Elixir). There are systems like Akka that simulate that to some extent on runtimes like JVM, but a cooperative scheduler and an actor model don't mix great. For non-actor model aspects I've been seeing more service level dataflow systems like KSQL / MaterializeDB gain traction, but are nevertheless a solution for read-models not application logic.

[jshen]

In short, making streaming architectures work require great central planing, a grand architect in the sky, or they fail miserably. I think that is an argument against streaming architectures ;)

[mrdoops]

I think that's a bit of a reduction/straw man to say they need a lot of architecture/central-planning. An individual developer can do it so long as they have access to domain experts to ask the right questions. Lack of proper planning will result in any architecture failing miserably - best not to code until we know what to code.

The implementation overhead as far as code-to-write for streaming architectures is comparable to CRUD. But there is less knowledge dispersed about practices on how to do it, so there is the cost of learning. It is more cutting edge after all.

[jshen]

It is admittedly an oversimplification for affect, but it’s more or less what you said. Maybe I’ve worked at bigger companies, we have over 1000 devs in my division, and trying to get them all aligned is very very difficult. I think central planning is a good analogy for that. The architecture that requires less central planning/coordination will likely do better in such a dynamic.

[mrdoops]

I expect streaming will be adopted like anything - first in smaller more mobile teams working in greenfield contexts, then later in large organizations when the idea is less novel and lower risk/cost to apply at scale.

Regardless I feel streaming has been around long enough now it should be urgently considered anywhere where the words "data pipeline" and "scale" are thrown around together frequently.

[fulafel]

There are many other applications that work well with graph / triple store style approach to data. See eg Datomic and the several Datomic inspired databases (like Crux, Datahike, Datalevin, etc) used as general work horses in the Clojure world.

I'd recommend trying non relational databases even if you're not sure.