[lobster_johnson]

Sure. I have been thinking about writing up a blog post about our work since it may be of use to others. A quick summary of some of our current microservices:

- Single sign-on. We have a service that abstracts users into identities that map to one or more actual logins. It facilitates the OAuth interaction with providers. To add login in any app, you just redirect the browser to the microservice's /auth/:provider API; it redirects through FB or whatever.

- For sites that don't want to rely on providers like FB, we also have a simple OAuth provider that has a basic user database with salted passwords, email/password management, etc.

- Data store. We have a structured, hierarchical document-oriented data store that is a thin layer on top of PostgreSQL. Most apps don't have their own database, but use this data store.

- Security. We have a common, path-based security model across apps that can be validated through calls into a central security microservice; it's mostly a registry of what apps delegate responsibility, and when you ask the security service about whether, say, you're allowed to modify the object at path acmecorp.blogapp.posts.1, it asks every app that is registered to be the authority for that path. For example, acmecorp.blogapp might use a role-based security system, but a different path exampleinc.cmsapp might use LDAP.

- Voting. For Reddit-style voting and kudos we have a small microservice.

- Search. We wrap our data store in an indexing system that uses ElasticSearch.

- Organizational management. We have a microservice that handles members, groups and their roles. It plugs into the security system so that we can delegate access to very precise parts of the system. For example, any user can see comments, only the author can edit a comment, and editors can update anything.

- Email/texting. We have a microservice that abstracts text and email, currently supporting things like Mailgun and Twilio. It allows us to swap out the implementation without the app knowing.

- GIS/map stuff. We have a separate microservices for geocoding and storing GIS features.

- Image/audio/video processing. We have a wrapper around ffmpeg and other tools that processes transcoding requests.

An important piece of the system is the extensive use of pub/sub via RabbitMQ exchanges to asynchronously publish and listen to events. For example, any microservice can listen to changes to any part of the data store (based on path or event). For example, our search microservice automatically populates ElasticSearch with content from the data store.

Lastly, every app has a microservice as its own backend; for example, if the app is a blog system, we have a frontend (eg., a Node.js desktop/mobile web UI), and a separate backend tailored to that frontend; the frontend treats the backend like any other microservice.

[jshen]

Thanks, this is really useful in comparison to my experiences. It all makes a lot of sense to me except the data store. Is the data store supposed to replace all direct database use by other services?

[lobster_johnson]

Not at all. We have apps that have their own private data stores either because they need complicated models with many inter-model connections, or because their data is completely private to themselves, and there is absolutely no benefit to a sharing it through an API.

The data store is intended primarily to be used for hetereogenous "content collections", for the lack of a better terms -- blog posts, comments, classified listings, products, articles, events, messages, etc. Anything that exists in bulk with simple CRUD semantics. It's less suitable for structural objects such as sites, users, groups, memberships, hierarchical, etc. where the semantics need to go beyond CRUD.

The data store model is extremely simple, and therefore somewhat poor in terms of possible interactions. There is very limited support for querying, for example, and no support for transactions. It essentially a slight step up from a key-value store. Interactions that require complicated queries need to go through other services such as ElasticSearch.

But there are benefits. One is that many apps simply don't need very complicated schemas. Another is that this sort of model forces you to optimize the schema in such a way that you avoid complicated joins. Thirdly, it's easy to integrate with a security system. Overall, it's just stupidly simple to get started. No setup, just start posting stuff.

However, we do see the need for a richer data store. A colleague mine is developing a new data store that is optimized for graph-like structures (probably layered on top of OrientDB), which may end up being a replacement for the current store because content can also benefit from being graph-like; blog posts have authors, for example. Another thing we have noticed is that the lack of schema validation leads to discrepancies, and it would be good to have this.