Hacker News new | ask | show | jobs
by jaseemabid 504 days ago
A notable example of a large-scale app built with a very similar architecture is ATproto/Bluesky[1].

"ATProto for Distributed Systems Engineers" describes how updates from the users end up in their own small databases (called PDS) and then a replicated log. What we traditionally think of as an API server (called a view server in ATProto) is simply one among the many materializations of this log.

I personally find this model of thinking about dataflow in large-scale apps pretty neat and easy to understand. The parallels are unsurprising since both the Restate blog and ATProto docs link to the same blog post by Martin Kleppmann.

This arch seems to be working really well for Bluesky, as they clearly aced through multiple 10x events very recently.

[1]: https://atproto.com/articles/atproto-for-distsys-engineers

3 comments

That blog post is a great read as well. Truely, the log abstraction [1] and "Turning the DB inside out" [2] have been hugely influential.

In a way this article here suggests to extend that

(1) from a log that represents data (upserts, cdc, etc.) to a log of coordination commands (update this, acquire that log, journal that steo)

(2) have a way to link the events related to a broader operation (handler execution) together

(3) make the log aware of handler execution (better yet, put it in charge), so you can automatically fence outdated executions

[1] https://engineering.linkedin.com/distributed-systems/log-wha...

Table/log duality goes back further than Kleppmann though. An earlier article that really influenced me was

https://engineering.linkedin.com/distributed-systems/log-wha...

Martin Kleppmann was also directly involved with Bluesky as a consultant.