[augustl]

Statistically (and from experience) I'm probably the weird one here, but I cannot fathom why Datomic isn't more popular.

I get that postgres is a good default in many cases, and I don't expect SQL to die tomorrow. But there are _so many_ apps (most/all backoffice apps I've worked on for example) with 10s or 100s of transactions per second at most, that would love to have the data available directly inside your business logic, and where both business logic and devops would improve by many orders of magnitude by having a full transaction log of all changes done to your data over time.

Is it _just_ because Datomic is different and people don't get it, and that preconceived notions makes you think Datomic is something it isn't?

Here's to the crazy ones!

[koito17]

Professional Clojure dev for some time. Here is what prevents me from using Datomic.

- no Docker image; still distributed as a tarball. Although com.datomic/local exists, it only provides the Client API, so it's mostly suited towards mocking Datomic Cloud.

- Datomic Cloud is designed around AWS; applications wanting to use Datomic Cloud must also design themselves around AWS

- Datomic On-Prem does not scale the same way Datomic Cloud does (e.g. every database has an in-memory index; the transactor and *all peers* need to keep them all in-memory)

- No query planner whatsoever. In databases like XTDB, the order of :where clauses doesn't matter since the engine is able to optimize them. In Datomic, swapping two :where clauses can transform a 10ms query into a 10s query.

In addition to the above four points, I strongly believe the following points prevent others from using Datomic.

- Writing Datomic queries outside of Clojure (e.g. Java) requires writing everything in strings, which feels awful no matter what.

- If you are not using a JVM-based language, then there is no choice but the REST API for interaction. The REST API is orders of magnitude slower than the official Clojure and Java clients.

- Too many tooling exists around the pgwire protocol and various dialects of SQL. Datomic obviously does not fit into either of these categories of tooling.

- Applications like DBeaver do not support Datomic at all. The best you can do is access the datomic_kvs table from a database using JDBC storage engine.

[refset]

> In databases like XTDB, the order of :where clauses doesn't matter since the engine is able to optimize them

Rich's observation was that query optimizers often get things wrong in ways that are hard to predict or control, but he's not fundamentally opposed to their use. That said, building a decent optimizer is a huge undertaking and I think they took the right decision to not attempt to bundle that sort of complexity into the original vision for Datomic otherwise they might never have shipped.

The state-of-the-art commercial engine for query optimization and execution algorithms in this 'triple' space is probably https://relational.ai/

[marcinzm]

As I see it, Rich's stance is that of an expert in the database that doesn't need to deliver business features using the database. New users are not experts and even experienced users that work for companies have pressure to deliver features. You can get initial popularity by targeting these types of expert users working on more experimental products. However long term growth and popularity requires targeting the other 99.9% of users. I've seen one company adopt Datomic due to this type of user and then a couple years latter rip it out because as it grew it's developers were no longer of this type.

[refset]

The original RDBMS vision was very explicitly for the users (both developers and analysts) to not have to be experts in their own database in order to achieve useful work, and without needing to think about procedural/3GL code from the get-go. In the intervening years query optimization has gotten a lot better, and hardware shifts have only worked in favour of this vision, but there's still a lot of work to be done before databases are truly "self-driving": https://www.cs.cmu.edu/~pavlo/blog/2018/04/what-is-a-self-dr...

Until that's the case I can understand why people are tempted to bypass this traditional RDBMS wisdom, especially if they have a very strong conception about their data models, access patterns, and need for scale (e.g. see also Red Planet Labs 'Rama').

[kolme]

If you don't have RDBMS wisdom and program an application around a RDBMS, your application is going to suck.

No ORM or framework is going to save you.

[closeparen]

In my experience, structuring a query to execute efficiency requires some basic software-engineering thinking about what's going on, while convincing a query planner to do the right thing requires deep expertise in the query planner.

[refset]

> convincing a query planner to do the right thing requires deep expertise in the query planner

The advantage is that that can be somebody else's job though, and ideally (eventually) an AI's job.

[panick21_]

I kind of agree, I find it was never that easy to set up and use. I also found the documentation to be quite limited on specific points I wanted to know.

They showed of a JOOQ like Java API for Clojure once but as far as I can see, this was never released. That is crazy to me, using it amazingly well from Java and friends would seem to me to be an absolute no brainer. That alone made it basically impossible to be adopted. Going from SQL/JOOQ to Strings was just not gone happen.

They focused so much on Datomic Cloud, and that just isn't where most people are gone deploy. Specially in the age of Kubernetes and Docker. Its kind of crazy that there were not official Docker images and things like that.

So even while I love Datomic conceptually, and once you have it set up with Clojure its pretty awesome. I would hesitate to really use it for a larger project.

I would really love if NuBank simply open-sourced it.

[cloogshicer]

As someone without Clojure experience but eyeing Datomic from the sidelines, thank you for the detailed answer, super interesting!

[tvaughan]

For an example of Datomic in containers, https://github.com/carrete/datopro-mic

[cfiggers]

Would you reach for XTDB instead of Datomic?

[JB024066]

>No query planner whatsoever. In databases like XTDB, the order of :where clauses doesn't matter since the engine is able to optimize them. In Datomic, swapping two :where clauses can transform a 10ms query into a 10s query.

Datomic has query-stats https://docs.datomic.com/reference/query-stats.html

[jwr]

Clojure developer here. I make a living from my SaaS, which is written in Clojure.

Reasons why I don't use Datomic:

* I've been burned in the past by a not-very-popular closed-source expensive database developed by a small company (specifically, Franz Inc's AllegroCache used with AllegroCL).

* I don't actually want to preserve all history. I am worried about performance.

* The data model doesn't fit my use case. I know my usage patterns and queries very well, so I am better served by KV stores.

* None of the storage engines is a good fit for me.

In case you wonder, I've been using RethinkDB and I am now moving to FoundationDB. I need a distributed KV store with strict serializability (https://jepsen.io/consistency/models/strict-serializable) consistency, running on my hardware.

In other words, not every database is a good fit for every application. The "just use Postgres!" crowd is wrong, and so is anybody who says "just use Datomic".

[Skinney]

Part of the reason is that it's been a commercial database until recently. I also think it hasn't helped that it's closely associated with Clojure (for natural reasons), which might make Java developers turn away from it.

It's a shame, though. Datomic is a marvel.

[kragen]

quite aside from whether or not it's 'commercial', it's proprietary

[amelius]

Yeah, the cost is the biggest reason probably. There's probably a lot of in-house developed Datomic clones out there (I know one, that was developed even before Datomic was a thing).

[ndr]

I love Clojure and would love to use Datomic. The fact that it is not open source pushes me away. The risk of being locked in is far too great for anything serious. There's no Valkey move if someone captures Datomic.

[Arjuna144]

Absolutely true!

"No Valkey move", I like it! That should be come it's own meme!

Being free but not open source makes no sense at all. It is just our little ego that wants to "keep" what we think of as "ours" to ourselves.

Give all you have to all and progress will go so much faster. Others may learn from the implementation of Datomic and make something better. So your ego is hurt, but humanity wins!!

Give all you have, and it will never be enough! Give anyway!

[Arcanum-XIII]

Pricing killed it for my experimental project.

Of course the complexity of having to set a cluster early on doesn’t help for small shops.

Then there’s the complexity of learning datalog - love it, but when I see people already struggling with SQL, I’m not confidant about using it generally !

[augustl]

It's free now - but I suppose 10+ years of being both closed source and paid didn't do wonders for adoption.

What part needs a cluster in your experience? The Datomic deployments I've been involved with have been running on a single server, with a single instance of backing storage etc.

Datalog is interesting indeed... Back when I first started using Datomic in 2012 I had just fundamentally decided to use it, and it took probably a week before the query language "clicked", i.e. to be able to actually compose my own queries and not just copy paste my way to something that works.

[kragen]

to clarify, it's not free as in 'free software'; it's free as in 'free beer'

[OJFord]

In reply to 'its pricing killed it for me', that was not ambiguous.

[intrepidpar]

Datatomic is not open-source, is it? And I think the licensing model was a bit odd. Those definitely don't help.

[sswezey]

Correct. It's free, but not open-source.

[pjmlp]

I guess because those of us that are fine with Datomic, also don't have any issues going into IBM, Oracle or Microsoft RDMS portfolio of database products.

[benrutter]

My best guess is Datomic's coupling to the JVM makes it great for clojure developers (or java, scala etc) but not even considered by python or javascript shops, which make up the vast majority.

If python and java has better interop, I think the story would be a little different maybe? My experience is that java is always a little stubborn as "child code" so libaries like pyspark often involve notoriously difficult set up.

[imhoguy]

Tabular data model, thus also relational, is in use since dawn of known civilization [0]. Anything what can't be described and literally touched as table causes discomfort and will do for centuries to go. This is why CSV is still the king in data transfer.

Data outlives logic. Simple data structures survive complex data structures.

And when you hardly tie data retrieval with complex logic and specific technology then one day you may not be able to simply retrieve it.

~~~

"ODE is a database system and environment based on the object paradigm. It offers one integrated data model for both database and general purpose manipulation. The database is defined, queried and manipulated in the database programming language O++ which is based on C++. O++ borrows and extends the object definition facility of C++, called the class. Classes support data encapsulation and multiple inheritance. We provide facilities for creating persistent and versioned objects, defining sets, and iterating over sets and clusters of persistent objects. We also provide facilities to associate constraints and triggers with objects. This paper presents the linguistic facilities provided in O++ and the data model it supports." 1989, [1]

Do you see similarities? It even started similarly as a research project. And it is gone now, both logic and data, likely gone with last program using O++, and will be with Clojure.

[0] https://www.datafix.com.au/BASHing/2020-08-12.html [1] https://dl.acm.org/doi/abs/10.1145/66926.66930

[panick21_]

And multiple table, connected make graphs. Almost nothing is a single table. If you want a single flat table in datomic, you can have it. In fact its literally just a big flat table conceptually.

> And when you hardly tie data retrieval with complex logic and specific technology then one day you may not be able to simply retrieve it.

You don't know how Datomic actually works do you?

[TacticalCoder]

> I get that postgres is a good default in many cases ...

And Datomic can run on top of PostGres anyway right? So it's not as there wasn't a lot of the knowledge that wasn't reusable: not for the queries themselves but everything about managing the database. Or am I wrong here?

[augustl]

Yes - if I understand you correctly :) Datomic writes opaque blobs of index data to existing storage, such as Postgres or dynamodb or a handful of others. But you can't query that data meaningfully directly, it's just a bunch of compressed datomic-specific index data in there, no domain structure or query power etc.

[emmanueloga_]

“Simple” comes from simplex (Latin for “not braided”), while “complex” means “braided together.” Datomic? Well, it feels like a whole lot of braiding going on. Datomic: complex made hard?

Datomic requires a storage backend (e.g., Cassandra, DynamoDB, MySQL, etc.), transactors, peers, and a separate transaction log... also, it appears the few real users of Datomic are primarily running it in AWS. Trying to run it elsewhere likely introduces more complications and uncertainties. Good luck getting support, too—you might have to wait for Rich Hickey to finish his hammock nap (ba-dum-tss)

While Datomic's immutability, time-travel queries, and Datalog query language are conceptually cool, they seem like niche features. My guess is that in 99% of the cases SQL can do the job just fine. Also, Rich Hickey may not have braids, but those curls come pretty close.

Without deep pockets for high-RAM servers, many tend to steer clear of Java. If Datomic were packaged as a single binary with fewer dependencies (and no JVM requirement), it could attract more users. As it stands, it's a complex setup with limited benefits, primarily suited for specific use cases, I think (financial auditing / historical data analysis?).

---

COLOPHON

(from the Greek word κολοφών [kolophōn], meaning "summit" or "finishing touch")

I don’t have real-world experience with Datomic; I played around with Datascript and ultimately steered away from Datomic after perceiving a low ROI given its complexity.

I hope my fellow fans of Clojure and the venerable 'Simple Made Easy' talk can forgive this little attempt at humor at Rich Hickey's expense! :-p

[augustl]

> Without deep pockets for high-RAM servers

I just got a dedicated server on Hetzner to test out some things. It's $70/month with 64gb RAM and a CPU that builds a complex C++ thingie in 11 minutes where my laptop spends about an hour :D

Scaling is of course not trivial, but the same set of backoffice apps I've worked with throughout the years that would be a good fit for Datomic, has a working set for the database much smaller than 64GB.

> they seem like niche features

That's the thing, though. Maybe it's because I've use Datomic a bunch. When I'm on projects that use a SQL db, a handful of problems are just fundamentally solved in Datomic, and none of the super knowledgeable devs that know SQL in and out are even aware that they are problems.

Some examples:

- What caused this column of this row to end up with this value?

- Oh no, we were down 3am but when the first person investigated it at 7am (oh those backoffice SLAs...) everything works fine and nobody knows that the db state was at 3am

- When we wrote value X, what other values did we also write at the same time?

- We need to carefully write a transaction that reads some data, writes some data, and reads some more data, and hopefully lock correctly, and hopefully we understood the isolation level (that we probably didn't set anyway) correctly and...

Which makes me think I must be the crazy one...

[emmanueloga_]

The easier of debugging does sound super appealing, the immutability and append-only design sounds really cool, especially if you don't really have that many transactions going on! I tried Terminus in this space, and there's also Dolt. [1]

How about the append-only/size side of things? I'm guessing if you use Dynamo or Cassandra, you basically forget about it (except when it comes to the bill ...). Is trimming the data straightforward if you don't have that much storage?

--

1: https://www.dolthub.com/blog/2022-03-21-immutable-database/#...

[dimitar]

Datomic Pro doesn't need AWS, I don't see what complications it can cause. It has some helpers for running it in AWS (like a bucket to export to CloudWatch), but none of them are mandatory. And if your use case is small you can run with dev storage (https://www.h2database.com/html/main.html); pretty much the same as you would SQLite.

It might be slightly harder to get started with, but then the simplicity comes in when it is time to solve common business problems. A trivial example would be - we have this nice db, now our clients want reports. You run your reporting as a separate clojure process, it doesn't impact production at all, without needing to setup reporting databases and log-shipping.

[watt]

"complex made hard" would be great. it seems you have missed the point of the talk: complex is always easy, complex is always where the gravitation pulls you towards.

[zefurocks]

curls jealousy

[dustingetz]

Rich said (in a random podcast, which was not widely circulated - possibly Cognicast) that their mission with Datomic Cloud was for it to be the easiest way to get started with clojure. He even used the word “Easy”! (unless i misremember, this needs to be checked). Anyway, Cloud had consumption based pricing and developers, companies, and even hobbyists are all accustomed to paying for managed cloud infrastructure. To me this seems like a viable strategy, and devs would have forgiven the previous licensing issues -- we just want to get paid to play with cool toys at work, you gotta work the money stuff out with the bosses.

The problem with Cloud was that it didn’t deliver on easy. It was not the Heroku they envisioned, rather it was a scary AWS amalgamation that required deep AWS knowledge to debug their CloudFormation templates to get it to work. This is not even really their fault!, as CloudFormation is a dumpster fire, there are much better cloud orchestration tools now. And while AWS's first few products were rock solid (S3, EC2, DynamoDB), around this time is when AWS turned on the enterprise growth machine and pivoted into box checking and everything cloud began to turn to quicksand underneath their feet.

On top of cloud quicksand, Datomic Cloud had technical scalability challenges which they brute force engineered through, motivating the architectural shift away from the easy synchronous/local entity api to the annoying async/remote client api, and then back again sorta with Ions -- but at the cost of, like, 3 years of product roadmap, during which time Typescript exploded in popularity, the serverless hype cycle began to break and SQL started making a comeback.

Also, the Java API had poor adoption. selling predominantly into Clojure commercial users isn’t a great market so consequently: no VC, small team, no sales/marketing/devrel - at a time when the startup boom was gaining momentum, developer infra was getting funded for the first time so competitors were spending tons of money on beautiful docs, marketing, full time twitter accounts, etc.

I still think there’s elements of an amazing product here, but the business window for a Clojure database has kinda closed - because Clojure itself has lost momentum. Datomic Cloud at its essence is a clojure hosting platform, the business is Clojure itself. I am very bullish on Clojure, I think it never found its killer app, and once found, once a money vector is opened, a bunch of cash (like, $100M†) will make all these problems go away and deliver on Clojure's mission to bring "simple made easy" to mass market application development, which is desperately needed. Maybe Hyperfiddle & Electric?

† Paraphrasing a Materialize press release, “it takes $100M to build a production ready database” -- and for the record, Materialize isn't doing too well, nobody seems to know what it is for and the VCs replaced the CEO earlier this year. Full quote: "Why did we raise $100M? Put quite simply, we believe this is the order of magnitude of investment that it takes to build a production-ready database. Databases are notoriously hard to get right, and we do not intend to cut any corners."

In the end, Cognitect chose to spend their lives doing something hard that matters, and I deeply respect that.

[panick21_]

> To me this seems like a viable strategy,

To me this seemed insane. The idea that I would use some pricey consumption based cloud think rather then using a systemd service or docker container locally to do most of my development is crazy. I don't want to deal with Amazon accounts authentication, VPC and all that other jazz just to start with a tiny project.

Even outside of the other stuff you mentioned.

> Also, the Java API had poor adoption.

They also put no effort into it. We use JOOQ, and I don't see why a JOOQ like API for Datomic wasn't doable. With the existing Java API, no wonder no Java shop would use it.

Not having a simple Docker container I can run and connect to a Spring Boot project within 10 minutes, so that my Java colleagues could use it, made it a a complete non-starter.

> In the end, Cognitect chose to spend their lives doing something hard that matters, and I deeply respect that.

Truly building something that really, really matters requires large adoption. And it seem to me every move they made was the opposite.

I can understand not going open-source, but honestly, to get really adoption, real wide traction, you need to be open and be well integrated into Java/PHP/JS and Python. And it seem to me they never really cared about that much at all.

[diggan]

> Truly building something that really, really matters requires large adoption. And it seem to me every move they made was the opposite.

> but honestly, to get really adoption, real wide traction

> they never really cared about that much at all.

It's all a matter of perspective. Rich been really upfront with that both Clojure and Datomic are products of Rich's solution to particular problems he experienced.

Datomic does really, really matter, even with the "small" adoption it has, for me. Even if I haven't used it myself a lot. And who are anyone of us to say what "truly matters" when it comes to how we spend our time? Clearly, Datomic does matter, otherwise these people wouldn't have spent a decade building it, so it does matter on some level.

Maybe that doesn't match up to your "truly, really, really matters" imagination, but it feels kind of weird to reach the conclusion that Datomic doesn't matter, based on what you believe to be impactful.

Ideas can live on beyond what the original projects carry, which is clearly the case with Datomic, and with basically any project (so far) Rich decided to work on.

[augustl]

> I still think there’s elements of an amazing product here

Interesting framing. The people behind Clojure and Datomic aren't known for being amazing at scaling and shipping products (i.e. marketing and all that jazz).

This is also not me judging them for it, I haven't built and shipped a Datomic, much less marketed it.

[dustingetz]

They shipped and scaled Clojure!