[CmdrDats]

So to have a concrete example: in our preliminary test of Datomic, we threw all our page view events directly at Datomic. Every single attribute in the event gets indexed at least 3 times and it grew out of a manageable scale really quickly. We had to find a solution where we can still maintain that data and aggregate it into Datomic when and where we can make useful sense out of it.

In certain cases you can definitely put the events directly into the transactions. In our case, it's just not the right fit and I suspect that many others will find the same constraints apply.

[jared314]

> Every single attribute in the event gets indexed at least 3 times and it grew out of a manageable scale really quickly.

Ok, so it was a technical issue. I assume you attempted to change the indexing.

The technical reasoning did not come through in the project page or your comments.

[lomnakkus]

It's not a technical issue. The fact that field "x" in a record changed from 4 to 5 (even if you can know every value that has ever been in that field) will never be able to tell you why that value changed.

Was it because a CSR got a call from a pissed-of-customer and agreed to change it? Was it because a third-party system fucked up and decided give the customer a little more "x"?

This is where Domain-Driven-Design comes in and CQRS/Event Sourcing really shines. The net effect of those two events I just described may be exactly the same for MY domain, but if there are other systems that are listening for the domain's events they may come to radically different conclusions. In CQRS/ES you would actually record those two things as different events, semantically.

It really upsets me that people don't get this about CQRS/ES. Maybe I should just start spamming links to Greg Young's talks...

[jared314]

I believe you misunderstand my comments.

A Datomic transaction is also an entity, with associated facts about that transaction. You can add custom facts, such as your application's domain event, directly to the transaction entity to further describe the event/transaction. You can then query over the transactions as you would any other data. In CQRS/ES terms, it is an event store with snapshots on every event without the cost of duplicate data.

The technical issue appears to be that their event stream was of a high enough throughput that the indexing caused a space issue.

[CmdrDats]

My apologies - my example was to illustrate a fit mismatch, not so much technical deficiencies.

Let's try it from a different angle: If you wanted to include every attribute of every event into Datomic as well as the aggregate view, you would have to add schema attributes for every event property AND the logical aggregate properties.

The extra schema doesn't really buy you anything other than being able to query your events, which you'll usually do by event type and date range. If you do need richer querying you're likely looking at an aggregate that happens to match the events. That's ok, and a design decision you can make.

Using DynamoDB raw means that events can have arbitrary shapes (including nested data structures) and you just dump them in verbatim. Then you only worry about your aggregate schema in Datomic.

In cqrs-server, we are also tagging the transaction with the event uuid, should you need to pull out the raw event from Dynamo.

[jared314]

> you would have to add schema attributes for every event property AND the logical aggregate properties.

In that case, you can serialize the hash-map with the fressian serializer (the one Datamic uses internally) and store it in an attribute of type :db.type/bytes. If the data is large (you didn't specify), then the link to an external data store is a good choice. The lack of an arbitrary shaped data type is a missing feature of Datomic, but is easy enough to work around.

I'm not sure Datomic is a mismatch. But, I don't know all the details of your situation. My main issue is with the lack of an explanation as to why you needed all those extra parts in the project page. The technical issues you solved, with those extra pieces, didn't come through in the project page.

[CmdrDats]

You raise a good point about not being clearer in the project page. I'll give it some thought and probably expand a bit on the dynamodb and datomic sections to incorporate the reasoning.

Again though, you are right, the events could be added to the datomic transactions with a fressian serialized byte array (we're fressian serializing into dynamo anyhow). It doesn't seem ideal to commit transactions with only the event data attached to the transaction and then at some later time possibly create an aggregate derivative from the event in a separate transaction.

The bottom line is: The event store and the transactional aggregate view are two very different things - putting both into one place conflates the two. We simply picked what we felt was better suited for each job, independent of one another. You could trivially point the event queue at Datomic instead - it really just doesn't buy you that much.

[lomnakkus]

Oh, I see. You're right -- I misunderstood. I guess I have a sort of reflex/twitch going on about CQRS/ES vs. Datomic. Mea culpa. :)

[dantiberian]

Datomic is a great event log, but it is explicitly not meant to be used in high write volume environments, as it is limited by the serial transactor.

[dj-wonk]

I think of Datomic as much more than a write log, since it includes the Datalog query language.

Also, how are you defining "high write volume"? Have you looked at metrics of how fast Datomic's serial transactor can be?

[dantiberian]

From the FAQ: "When is Datomic not a good fit? Datomic is not a good fit if you need unlimited write scalability, or have data with a high update churn rate (e.g. counters). And, at present, Datomic does not have support for BLOBs."

I agree Datomic is far more than a write log but in the context of this thread, the discussion was about write logs.

[sanderjd]

Interesting. Couldn't you achieve the same thing by aggregating data into an RDMS "when and where [you] can make useful sense of it"?