Hacker News new | ask | show | jobs
by alexgunnarson 675 days ago
We employ a knowledge graph at Deft (https://shopdeft.com) to enable searches over ~1M products, amounting to about 1B triples. Because of the complexity of the queries involved, the expressiveness of our data model — supporting n-ary/reified relations, negation, disjunction, linguistic vagueness, etc. — and our real-time latency targets, we built a graph DB engine "from scratch" (certain components are of course from open-source projects). Even RedisGraph wasn't fast enough for the purpose; ours (Deftgraph) is 700x faster on our queries thanks to some SOTA optimizations from various recent papers. You'll notice on our site that the overall search latency is generally acceptable but not great; the vast proportion of that latency comes from 1) LLMs and 2) a less-optimized other graph DB, Datomic, that we still store some of our data in for legacy reasons.

LLMs are great, but knowledge graphs are IMO indispensable to tame their shortcomings.
2 comments
traverseda 675 days ago
If you have a graph database that is 700x faster on real world use cases than the next nearest competitor, why aren't you selling it? Given the current AI gold rush, it seems like a no brainer to get some VC cash, hire some sales people, and start selling shovels.
link
j-pb 675 days ago
I'd love to hear about your absolute numbers.

We had a similar problem, Datomic/Datascript not having an open format like RDF, but RDF being clunky and slow, so we build our own open-source solution in Rust (https://github.com/triblespace).

On an M1max we're currently at ~3us per query for a single result (so essentially per query overhead), and have something like 1m QRPS for queries with 3-4 joins.

I'm curious if you've somehow managed to shave off another order of magnitude, as I suspect that most WCO joins will be similarly limited by memory bandwidth. We for example worked out a novel join algorithm family (Atreides Join) and supporting trie based in-memory and succinct zero-copy on-disk data-structures, just to get rid of the query optimiser and its massive constant factor.

link