[swyx]

we use event sourcing under the hood for its fault tolerance and scalability (and tracing/observability). it’s abstracted away for you by our SDKs.

in other words when you use us you get the main benefits of ES without the downside of having to code it up yourself, which is a common pitfall of homegrown ES systems.

[MaknMoreGtnLess]

That's fair but I have a question:

If I DID want to get access to the event sourcing mechanisms under the hood - can I?

My hypothesis is that I can use Temporal to implement high quality DDD pipelines that follow CQRS - having explicit access to the event sourcing mechanisms would be fantastic!

Otherwise, it's an issue as most DBs can be argued to use some form of event sourcing mechanisms under the hood but they arn't built to be exposed outside the DB - hence an event sourcing mechanism has to be reimplemented (hence pitfall of homegrown ES systems).

What are your thoughts?

[swyx]

you could, by directly accessing the backing db, but we dont encourage that. The other way you could do it is read-only access by polling our event history APIs - that would be doable.

i try to say "we use event sourcing under the hood" rather than "we do event sourcing for you" for this reason - if you use us and expect the same level of control as homegrown youre gonna be disappointed

[MaknMoreGtnLess]

understood.

What's the design pattern you recommend Temporal users use to implement multiple consumers?

Usecase: Handle a large, spiky backlog by dynamically sharding the consumer input, starting and putting multiple consumers to work until the backlog is within nominal specs.

Eg: Shipping orders during a sales event. Each S&H order takes the same response time but multiple S&H orders can be parallelized and there are a few hundred thousand orders pending. Of course, S&H order can fail, in which case they are restored to the pending state after a timeout or loss of a lock

[swyx]

ah for this one you dont need multiple “consumers”, you need multiple workers, in our model. they’re kinda the same thing when u get down to it but this is the paradigm shift.

every temporal workflow goes into an assigned task queue, and Temporal handles distributing/load balancing to multiple workers polling that queue. you can have 10000 orders coming in simultaneously to 1 task queue being processed by 5 workers for example and Temporal would register heavy load but would still process through all that work in due time. the beauty is that when you write workflows you dont have to worry about acquiring locks or whatever, just write as though you had one durable long running process per order. its very freeing.

this is not to say it does everything, ie Temporal is not a replacement for a true pubsub model. we just described N workers processing 1 type of workflow initiated 10,000 times, but its not designed for 1 event type initiated 10,000 times that are reacted to by N types of processes that are supposed to be completely decoupled.