[straws]

A number of years ago, I worked on a team (~20 engineers in total) that successfully carved off two relatively independent portions of a large Rails app using engines. I'm happy to see that Shopify is also using that strategy.

I'm curious to know more what sorts of challenges they have around managing dependencies across engines — I think what we were doing was fairly vanilla Rails, and we didn't have the opportunity to run into those sorts of issues.

[exterm]

The answer to that question could probably fill another blog post :D

Long story short, Rails and dependency inversion equals lots of friction. The whole framework is built on the assumption that it's OK to access everything from everywhere, and over the years we've built lots of tooling on top of those assumptions.

E.g. we heavily use https://github.com/Shopify/identity_cache with active record associations that cross component boundaries.

We also have a GraphQL implementation that is pretty closely coupled to the active record layer and _really_ wants to reach into all the components directly.

All of those problems can be overcome, but this is definitely an area where we have to working against "established" Rails culture, and our own assumptions from the past.

[straws]

I hope to hear more in the future!

Do you envision any extension points to the way engines are implemented that could better enforce boundaries? In our engines, there was nothing that referenced another engine's resources, leaving the main application to handle route mapping and ActiveRecord associations between app models and engine's models.

I feel like the use-case for engines has long been around supporting framework like functionality (Devise, Spree, etc), but I wonder if there are changes to be made that better support modularization for large apps.

[exterm]

> extension points to the way engines are implemented that could better enforce boundaries

Can you expand on that? I'm not sure I follow.

[sandGorgon]

What's the difference between "componentization+engines" and microservices?

From a deployment perspective are your engines deployed and scaled independently?

[exterm]

components are

- same database - same runtime - same deployment - same repository

That said, I don't think this is an either/or. It's a spectrum. you can have components within the same runtime and repository that have separate databases, or components that are using the same database but live in separate repos, etc.

From one monolithic app towards fully separated microservices is a spectrum, and I think developers should be enabled to move freely around that spectrum.

[sandGorgon]

I think components are the better option. Because it allows for separation of concerns without introducing deployment ...or worse : political complexity.

I call them Micro-SDKs.

[JohnBooty]

I worked on a large Rails monolith a few years back with a similarly-sized team and we took the "components+engines" approach too.... and it was a bit of a nightmare, honestly. It sort of felt like the worst of both worlds, relative to monoliths or microservices.

I strongly suspect, but cannot prove, that we would have been better off simply transitioning to "macroservices" -- breaking the monolith up into several (as opposed to dozens) of reasonably sized pieces.

• We were encouraged to componitize everything. When I left, we were up to a few dozen components, and the number was climbing rapidly. I'm not sure if the approach itself was the problem, or if the flux during the transition period was the real pain point.

• We had no real enforcement of interfaces between components. It was so easy to break things in other peoples' components.

• Theoretically that breakage would be caught by tests. But to catch that breakage, you needed to run the complete test suite (30-60 minutes) rather than simply testing your own component

• Essentially, it felt like we were suffering all the disadvantages of microservices, with the exception of coordinating deployments; from a devops perspective it was still just a single monolithic deployment

• We still had many of the problems associated with monoliths, such as slow deployments, long test suite times, and extremely high per-instance RAM usage

• Various small tooling and debugging issues related to using Rails but going too far "off the Rails"

I'm looking forward to digging into the linked article and learning how Shopify solved those issues. They seem to have quite a bit of engineering firepower at their disposal. Our management did not allow us to dedicate a lot of resources to internal engineering concerns like this.

(We essentially had one guy figuring it all out himself, and due to internal politics he was forbidden from considering a microservices or "macro services / multiple monoliths" approach. He was talented and did the best he could, considering)

[exterm]

> When I left, we were up to a few dozen components, and the number was climbing rapidly.

I should have included this in the blog post: The number of components _needs_ to be kept small. Shopify's main monolith is 2.8 million lines of code in 37 components, and I'd actually like to get that number _down_.

I like to compare this to the main navigation that we present to our merchants. It's useful if it has 8 entries. It's not useful if it has 400.

In a way, components are the main navigation to our code base. A developer should be able to look at what's in our "components" folder and get a general impression of what the system's capabilities are.

[JohnBooty]

That's an excellent (and hard-earned, I'm sure!) insight. Thank you.

    I like to compare this to the main navigation that we 
    present to our merchants. It's useful if it has 8 
    entries. It's not useful if it has 400.

Yeah, we essentially wound up with a "junk drawer" of components. I could see a lot of companies, like ours, making that mistake -- turning all the things into components.

As you said in the article, one of the benefits of components for you was that it truly forced you to think about a proper separation of concerns. In hindsight, that's an area where we really missed the mark for a variety of reasons, some methodology-related.

We practiced a rather strict version of Scrum. Management paid a lot of attention to our velocity from week to week: we needed to rack up those story points.

But, outside of the tiny team dedicated to the component effort, there were no story points to be had for supporting that effort. Therefore we were in fact incentivized not to support it. I remember one sprint where I did some refactoring work in order to achieve a better separation of concerns. It negatively affected our velocity for the week and that was noticed.

So, we were receiving a schizophrenic message from management. We were all to support the component effort.... but on our own time, apparently?

[kogus]

This is tangental, but I want applaud the mindset that produces the phrase "the opportunity to run into those sorts of issues".

[straws]

I meant it more as a testament to how far you can get with a Rails app before needing to consider using the power tools :^)