[PaulHoule]

I have been involved with graph databases and always thought Neo4J was a joke, it was based on an execution model which is not scalable at all. I heard story after story from people who tried it for projects which were just too big and wanted to ask, “what did you think would happen?”

[Railsify]

what about the execution model is not scalable? I've got a side project using neo4j, maybe I can avoid a major mistake here?

[rswskg]

You can't shard it. I think you can now maybe, but last time I looked you could only scale veritically, not horizontally

[tuyguntn]

I think sharding graph models are actually difficult problem.

[alostpuppy]

Any graph dbs that so scale?

[jandrewrogers]

Not really, but that depends on your definition of “scale”. To make one that scales well, you’ll need to solve a few difficult computer science problems that conventional database architectures don’t need to consider and therefore don’t address. These are problems like graph cutting, multi-attribute search without secondary indexes, cache-less I/O schedulers, and a couple others. Scalable solutions to all of these problems exist independently but I’ve never seen any graph database implementations that even attempt to address most of these, never mind all of them, and you kind of need to remove these bottlenecks.

As long as most graph databases are just a layer of graph syntactic sugar sprinkled on top of a conventional database architecture, they won’t scale.

[HyperSane]

Horizontaly scaling graph DBs is incredibly hard.

[holler]

What's your thought on AWS Neptune?

From the marketing page below:

"Scale your graphs with unlimited vertices and edges, and more than 100,000 queries per second for the most demanding applications. Storage scaling of up to 128Tib per cluster and read scaling with up to 15 replicas per cluster."

https://aws.amazon.com/neptune/

[HyperSane]

That isn't real horizontal scaling, they are just doing a single vertically scaled writer and read only replicas, just like they do with RDS. It probably is using the same infrastructure for it.

[rrwright]

1,000,000 events per second with https://Quine.io

https://www.thatdot.com/blog/scaling-quine-streaming-graph-t...

Full disclosure: I work on this project.

[jandrewrogers]

When I first saw the benchmark result I was pleasantly surprised by the performance, you rarely see that on a single large server but it is achievable if the implementation properly does all the hard bits.

Then I saw that it required 140(!) servers to achieve that result and now I’m wondering what all that hardware is actually doing. On a per-server basis, that is very low throughput, even for graphs. Efficiency that low will make it uneconomical for most graph applications.

[rrwright]

That’s not an optimized benchmark, just a demonstration using a real customer workload. Throughput depends on the workload but goes up to about 20,000 events per second per machine. All this is while simultaneously querying the graph and streaming out 20,000+ events per second. All that includes durable storage.

Price it out against Neptune instead and Quine is much less than 1% of the cost.

[lolive]

On a previous project, we reached the limit of 32 billions of unique IDs (Neo 3.2 if I remember well). And had to wait for the next version so we could add more data.

I left the project. But, as far as I know, there are still planes maintained and authorized to fly in the sky, so I suppose the DB is still up in production.

[lolive]

And I am pretty sure the number of IDs in the DB has skyrocketed since that time.

[logicalmonster]

JanusGraph may or may not fit your needs, though there's quite a learning curve even figuring out how to set it up.

[kleinsch]

Tao

[tszming]

so what graph database you end up with?

[PaulHoule]

It's been a long strange trip.

I don't think a single product is going to satisfy everyone and that's a problem with the category. If by "graph database" you mean you want to do completely random access workloads you are doomed to bad performance.

There was a time when I was regularly handling around 2³² triples with Openlink Virtuoso in cloud environments in an unchanging knowledge base. I was building that knowledge base with specialized tools that involved

   * map/reduce processing
   * lots of data compression
   * specialized in-memory computation steps
   * approximation algorithms that dramatically speed things up (there was a calculation that would have taken a century to do exactly that we could get very close to in 20 minutes)

Another product I've had a huge amount of fun with is Arangodb, particularly I have used it for applications work on my own account. When I flip a Sengled switch in my house and it lights up a hue lamp, arangodb is a part of it. I am working on a smart RSS reader right which puts a real UI in front of something like

https://ontology2.com/essays/ClassifyingHackerNewsArticles/

and using Arangodb for that. I haven't built apps for customers with it but I did do some research projects where we used it to work with big biomedical ontologies like MeSH and it held up pretty well.

I came to the conclusion that it wasn't scalable to throw everything into one big graph, particularly if you were interested in inference and went through many variations of what to do about it. One concept was a "graph database construction set" that would help build multi-paradigm data pipelines like the ones described above. One thing I got pretty sure about was that it didn't make sense to throw everything into one big graph, particularly if you wanted to do inference, so I got interested in systems that work with lots of little graphs.

I got serious and paired up with a "non-technical cofounder" and we tried to pitch something that works like one of those "boxes-and-lines" data analysis tools like Alteryx. Tools like that ordinarily pass relational rows along the lines but that makes the data pipelines a bear to maintain because people have to set up joins such that what seems like a local operation that could be done in one part of the pipeline requires you to scatter boxes and lines all across a big computations.

I built a prototype that used small RDF graphs like little JSON documents and defined a lot of the algebra over those graphs and used stream processing methods to do batch jobs. It wasn't super high performance but coding for it was straightforward and it was reliable and always got the right answers.

I had a falling out with my co-founder but we talked to a lot of people and found that database and processing pipeline people were skeptical about what we were doing in two ways, one was that the industry was giving up even on row-oriented processing and moving towards column-oriented processing and people in the know didn't want to fund anything different. (Learned a lot about that, I sometimes drive people crazy with, "you could reorganize that calculation and speed it up more than 10x" and they are like "no way", ...) Also I found out that database people really don't like the idea of unioning a large number of systems with separate indexes, they kinda tune out and don't listen until you the conversation moves on.

(There is a "disruptive technology" situation in that vendors think their customers demand the utmost performance possible but I think there are people out there who would be more productive with a slower product that is easier and more flexible to code for.)

I reached the end of my rope and got back to working ordinary jobs. I wound up working at a place which was working on something that was similar to what I had worked on but I spent most of my time on a machine learning training system that sat alongside the "stream processing engine". I think I was the only person other than the CEO and CTO who claimed to understand the vision of the company in all-hands meetings. We did a pivot and they put me on the stream processing engine and I found out that they didn't know what algebra it worked on and that it didn't get the right answers all the time.

Back in those days I got on a standards committee involved w/ the semantics of financial messaging and I have been working on that for years. Over time I've gotten close to a complete theory for how to turn messages (say XML Schema, JSON, ...) and other data structures into RDF structures and after I'd given up I met somebody who actually knows how to do interesting things with OWL, I got schooled pretty intensively, and now we are thinking about how to model messages as messages (e.g. "this is an element, that is an attribute, these are in this exact order...") and how to model the content of messages ("this is a price, that is a security") and I'm expecting to open source some of this in the next few months.

These days I am thinking about what a useful OWL-like product would look like with the advantage that after my time in the wilderness I understand the problem.

[Jupe]

Fun read above - very descriptive and interesting... thanks for sharing!

OWL? RDF? Were you an RPI graduate perhaps? (I wasn't but did visit them once as part of research project).

At the end of the day, triple stores (or quad stores with providence) never quite worked as well as simple property graphs (at least for the problems I was solving). I was never really looking for inference, more like extracting specific sub-graphs, or "colored" graphs, so property attribution was much simpler. Ended up fitting it into a simple relational and performance was quite good; even better than the "best" NoSQL solutions out there at the time.

And, triple stores just seem to require SO MANY relations! RDF vs Property Graphs feels like XML vs JSON. They both can get the job done, but one just feels "easier" than the other.

[carterschonwald]

Where are you based? Id love to hear more about this set of adventures over tea or coffee sometime

[PaulHoule]

Check my profile and send me an email.

[estro0182]

This is interesting. What is OWL?

[Jupe]

I think he's referring to Web Ontology Language; IIRC it is a kind of schema for relations in graphs. It was a big part of the Semantic Web surge from 10+ years ago.

[lolive]

RDF is the JSON of graphs. OWL is the Json-Schema of RDF.

[throwoutway]

Asking “what did you think would happen” is arrogant. They probably did not know as much as you

[teej]

If you pick a technology to deploy within a company, you need to own the outcomes of that choice.

“I didn’t know” isn’t an acceptable excuse. When faced with unknowns, it’s your job to anticipate, mitigate, uncover problems, etc.

No one can know everything. That’s a fact. But these issues with Neo4j aren’t exactly hard to find. There are loads of folks who have talked about their negative experiences with it. Setting up a proof of concept would confirm them.