[cebert]

Is anyone else considering using DyanmoDB for event sourcing instead of Kafka? We have a project at work where we store events into an event table. We then had a stream that invokes a distributor lambda. The distributor lambda lookups subscribers to a given event type. For each subscriber for a given event, we place a copy of the event to a SQS queue for the event subscriber. Each subscriber lambda can process events from its own queue. If processing fails, each subscriber will retry processing a configured number of times before moving the failed event to a dead letter queue.

This approach has been working well for us without the need of Kafka with our serverless apps. I was curious if anyone is doing something similar.

[mtberatwork]

Maybe I'm missing something but this setup sounds like a re-creation of Beanstalkd. https://beanstalkd.github.io/

[cebert]

I haven’t heard of Beanstalkd before. I am just starting to read there site, but this sounds like a distributed job processing system? There’s some potentially some overlap. Our goal was to have a serverless form of event sourceing. We don’t have 1000s of subscribers to events (we aren’t FB), so our approach is working ok for us. Thanks for informing me about Beanstalkd, I will read up on it more.

[mtberatwork]

If you look at the concept of "tubes" in beanstalkd, it seems to be quite similar to what you are doing in AWS. Each of your independent queues is just a "tube" in beanstalkd. Quite honestly though, you can do this same functionality with just about any message broker. No need to run all these separate services.