[lobster_johnson]

Looks nice. I particularly like the ORM, which seems well-designed compared to ActiveRecord.

That said, I have a distinct feeling this is too little, too late. I and my teams have long migrated away from frameworks, to microservices, or what some people call SOA, and we are not looking back. I strongly encourage this design over the old monolithic Rails approach.

These are mostly small, highly specialized Sinatra backends that speak data only through REST APIs. For such backends, you don't need views, and you hardly need models; when "deconstructing" your data model into multiple backends, the data model for each microservice often becomes absurdly simple. Mostly you need a simple way to express REST endpoints. Sinatra is very simple and doesn't support any abstraction, but that's mostly a good thing. And you can split your API across multiple files easily. The backends are getting so simple that it's starting to look very tempting to migrate to Go, where we could get real performance and concurrency.

On the front end, I would still use Sinatra if it were a classic app -- you can cobble together a "railsy" app with Sinatra by picking your template system, database layer and so on -- but mostly these days we build Node.js single-page apps that are sewn together with NPM and Browserify. Web app development has never been as simple and elegant as it is when done this way.

[fredwu]

The ORM part, largely due to the repository pattern, is very similar to my earlier attempt - Datamappify - https://github.com/fredwu/datamappify

Given time, effort and popularity, I could see Lotus::Model to become the de facto alternative to ActiveRecord, overtaking the ROM (Ruby Object Mapper, previously known as DataMapper) project. :)

[shime]

This sounds very interesting. Please make a blog post about this if time permits. We seriously lack good resources about micro services when it comes to Ruby.

[jshen]

Can you give done examples of the types of things these micro services do?

[stevekemp]

Not the poster, but I've written a few small sinatra-endpoints in my time.

For example I wrote a self-hosted comment-system, which is like a very basic disqus service. The back-end is a simple sinatra service that just allows "GET /comments" and "POST comments". (http://github.com/skx/e-comments/)

Last week I wrote a webhook-consumer, again using Sinatra. It receives a POST from github, parses the received JSON, and adds some details of the body into a queue for later processing.

Both services allow simple deployment and total may 50 lines of code each.

[prawks]

Sounds very similar to the things I've done with web.py in Python-land. Very small, lightweight services that had no need for the power of Django. One such example was a GET/POST/DELETE service to vote on submitted items (ideas, in this case).

[lobster_johnson]

Sure. I have been thinking about writing up a blog post about our work since it may be of use to others. A quick summary of some of our current microservices:

- Single sign-on. We have a service that abstracts users into identities that map to one or more actual logins. It facilitates the OAuth interaction with providers. To add login in any app, you just redirect the browser to the microservice's /auth/:provider API; it redirects through FB or whatever.

- For sites that don't want to rely on providers like FB, we also have a simple OAuth provider that has a basic user database with salted passwords, email/password management, etc.

- Data store. We have a structured, hierarchical document-oriented data store that is a thin layer on top of PostgreSQL. Most apps don't have their own database, but use this data store.

- Security. We have a common, path-based security model across apps that can be validated through calls into a central security microservice; it's mostly a registry of what apps delegate responsibility, and when you ask the security service about whether, say, you're allowed to modify the object at path acmecorp.blogapp.posts.1, it asks every app that is registered to be the authority for that path. For example, acmecorp.blogapp might use a role-based security system, but a different path exampleinc.cmsapp might use LDAP.

- Voting. For Reddit-style voting and kudos we have a small microservice.

- Search. We wrap our data store in an indexing system that uses ElasticSearch.

- Organizational management. We have a microservice that handles members, groups and their roles. It plugs into the security system so that we can delegate access to very precise parts of the system. For example, any user can see comments, only the author can edit a comment, and editors can update anything.

- Email/texting. We have a microservice that abstracts text and email, currently supporting things like Mailgun and Twilio. It allows us to swap out the implementation without the app knowing.

- GIS/map stuff. We have a separate microservices for geocoding and storing GIS features.

- Image/audio/video processing. We have a wrapper around ffmpeg and other tools that processes transcoding requests.

An important piece of the system is the extensive use of pub/sub via RabbitMQ exchanges to asynchronously publish and listen to events. For example, any microservice can listen to changes to any part of the data store (based on path or event). For example, our search microservice automatically populates ElasticSearch with content from the data store.

Lastly, every app has a microservice as its own backend; for example, if the app is a blog system, we have a frontend (eg., a Node.js desktop/mobile web UI), and a separate backend tailored to that frontend; the frontend treats the backend like any other microservice.

[jshen]

Thanks, this is really useful in comparison to my experiences. It all makes a lot of sense to me except the data store. Is the data store supposed to replace all direct database use by other services?

[lobster_johnson]

Not at all. We have apps that have their own private data stores either because they need complicated models with many inter-model connections, or because their data is completely private to themselves, and there is absolutely no benefit to a sharing it through an API.

The data store is intended primarily to be used for hetereogenous "content collections", for the lack of a better terms -- blog posts, comments, classified listings, products, articles, events, messages, etc. Anything that exists in bulk with simple CRUD semantics. It's less suitable for structural objects such as sites, users, groups, memberships, hierarchical, etc. where the semantics need to go beyond CRUD.

The data store model is extremely simple, and therefore somewhat poor in terms of possible interactions. There is very limited support for querying, for example, and no support for transactions. It essentially a slight step up from a key-value store. Interactions that require complicated queries need to go through other services such as ElasticSearch.

But there are benefits. One is that many apps simply don't need very complicated schemas. Another is that this sort of model forces you to optimize the schema in such a way that you avoid complicated joins. Thirdly, it's easy to integrate with a security system. Overall, it's just stupidly simple to get started. No setup, just start posting stuff.

However, we do see the need for a richer data store. A colleague mine is developing a new data store that is optimized for graph-like structures (probably layered on top of OrientDB), which may end up being a replacement for the current store because content can also benefit from being graph-like; blog posts have authors, for example. Another thing we have noticed is that the lack of schema validation leads to discrepancies, and it would be good to have this.

[kidmenot]

+1, I would love to take a look at an example or two, as this sounds very interesting.