[Jugurtha]

One of the best, and first, things we did when starting our machine learning platform was to design it using a plugin architecture. There's a lot of scar tissue and horrible experience through our previous ML products we built for enterprise.

Namely, it was extremely hard to onboard new developers to work on the product. They had to understand the whole thing in order to contribute.

Changing something was also hard, since it was intertwined. Adding a feature or removing a feature was hard. Especially given the fact we were not designing to spec, and the domain experts we were building for could not give feedback. That was a constraint outside of our control, so we were building for users we never met based on what we thought would make sense.

The first commits on our own platform were to establish a plugin architecture. There's a core, and there are plugins. We could add or remove functionality changing a config file. Applications are plugins and onboarding is easy and smooth, since a junior developer can start working on one plugin and then expand their knowledge.

We're reaping the rewards of that.

[yreg]

A way to tackle this further is to use an analytics tool like CodeScene. You link it with your repo and it shows you what people have knowledge across which parts of code[0].

Then you can identify parts that share too few contributors and encourage people to work on them together. You might also find parts where the knowledge is already lost, since everyone who worked on it already left the company. In that case some team can volunteer to take ownership of the code and take time to make some sense of it.

Of course, easier said than done.

[0] https://codescene.io/projects/167/jobs/55946/results/social/...

[daemin]

That kind of looks useful, would be nice if it handled Perforce and could be installed on private servers.

Don't think the company would want to release all the code publicly.

[yreg]

Are you sure it couldn't be installed locally?

My client is a bank with private GitHub servers, so I assume their codescene is self hosted as well.

edit: Yep, there is an on-premise option: https://codescene.com/pricing/

[nephrenka]

Yes, there's an on-prem version of CodeScene that can be run on private servers. The latest release is described here: https://codescene.com/blog/architectural-analyses-simplified...

To analyse Perforce, you setup an automated conversion to a read-only Git repo that you then point the analysis to.

[corytheboyd]

Fully support those who want to use these tools, just don’t forget they are not a crutch. You can still write hard to work with code and know exactly who the SME is. That doesn’t make the code less difficult to work with, less error prone to change. That can only be solved by design and enforced through discipline.

[nephrenka]

Fully agree! That's why I recommend to monitor the code health trends continuously -- the earlier we can catch any potential issues, the better: https://codescene.io/projects/167/jobs/55946/results/warning...

We have these code health trends supervised as part of our pull requests and use them as (soft) quality gates. [1]

[1] https://codescene.com/blog/measure-code-health-of-your-codeb...

[yreg]

Absolutely. This only helps to nerf the knowledge loss from teammates leaving the project, nothing more.

[Jugurtha]

And nerf knowledge loss from ourselves after a couple of [weeks/months/years].

[Cthulhu_]

Definitely; the biggest challenge in software development is keeping things manageable. I think this is why microservices are so popular; the individual services are manageable. But that's a misconception, because the overall architecture becomes a monstrosity, especially if there is no oversight (which in my experiences with the pattern, there wasn't).

[ownagefool]

I think microservices are and aren't an overreaction.

If you think of microservices as contract enforcement, it should be harder to produce unintended consequences on the macro level, because everything should flow through the api. (assuming you don't have something weird like several microservices manipulating the same data sources directly), so architecturally it's easier to understand code/data flows than the same monolith system that hasn't properly enforced modularity.

The main problem is most folks rushing head first into the silver bullet don't understand is things have trade off, and in microservice land it's versioning, testing, deploying, and monitoring.

Nowadays, though the tooling is pretty good, and if you put your microservices in a monorepo (seems backwards, I know) you can avoid the versioning, testing and deployment difficulties, and use GKE + istio and you've got tooling to help handle the ops problems, so actually, maybe enforcing code quality is actually the harder problem, and limiting the size and scope does sorta make sense.

[arkitaip]

Are there any drawbacks of using a plugin centric approach? Typically there is loss of expressiveness in code, loss of performance or disconnect between core and plugin development.

[dehora]

> Are there any drawbacks of using a plugin centric approach?

Here are some:

1. You're perhaps more subject to Hyrum's law. If plugin devs can see it, they will use it. The general observation here is that it's harder to control the visible interfaces and implicit dependencies you export than the dependencies and interfaces you rely on. As one example, semantic versioning doesn't cater for this at all. Plus, most of the practice knowledge in software is on managing relied on dependencies.

2. Dog follows tail. It can happen that a plugin becomes so successful the overall system evolution slows down. The core system can upgrade, but adoption/deployment can be constrained when a particularly valuable plugin doesn't move up to the latest and the customer base sees more value in the plugin than its core platform. This can compound poorly over time, and in extreme cases the desirable plugin can become its own platform/system (something I think business savvy tech leaders are increasingly aware, and wary, of).

3. Operational complexity. It can be harder to run and maintain a plugin based system than a closed one. 2 is a consideration here, but so are other concerns, such as security and resource isolation. Strategies vary, but who pays this cost on a relative basis is one of the more (and perhaps the most) interesting aspect of working on or using plugin systems. As one example of this, think about allocating responsibility for bugs.

4. R&D complexity. It may take more time to design and build a plugin system than a closed one. Incrementally evolving to a plugin system can be difficult if you didn't start there to begin to with. So you usually need a clear opening motivation to delay reward (or avoid over engineering) to invest in a system design where functionality can be extended by non-core developers.

[Jugurtha]

Not that I could see at this point. It is not a panacea, but for us, it was a way to contain scope at different levels.

For exaple, we have the platform and it has icons on the sidebar for Notebook, Object Storage, etc.

Every single one of these is a separate application and a separate repository. These applications are independent in how they deal with business logic, so there's no loss of expressiveness. They just must present certain "receptors" or interface if they want to be plugged into the system. The "interface" is a big word, and someone can produce a valid minimal plugin (that does nothing except be loaded) in two minutes.

This allows us to contain details of a plugin to the plugin itself, and not having it leak to other parts of the product. If we want to activate/de-activate the plugin, it takes less than 10 seconds manually.

Now, sometimes a plugin depends on another plugin. But they make their requests to that plugin, and fall-back to something else in case that plugin is unavailable.

The amount of engineering time this has saved us is delightful. I think of all the code we did not have to write and it makes me smile.

That's for containment and encapsulation at the application level. But we also follow that mode at the functionality level, too. For example, model detection and tracking is done by plugins.

We like to do things and have an abstraction so that we can churn out functionality for similar things "industrially", without thinking too much, but also so we could remove things easily without breaking the rest. Making code not just easy to add, but easy to remove is important. When we did that, we were able to remove a lot of code, too.

It is a spectrum, and we started by using it to contain at the "app" level.

[marktangotango]

That’s a rosy picture, but I’d point out that atrocities such as Eclipse Rich Client platform and associated OSGi specs are where this plugin concept can lead and it has its own problems! Complexity and discoverability are two!

[Jugurtha]

It's not a panacea, but as a guiding thought it has served us well. We don't go all in on things "just because", and we develop what we need as opposed to what we imagine we need, and stop at an abstraction level that gets the job done.

[arkitaip]

Thank you, I appreciate the detailed answer.

[leetcrew]

you need to have someone who is very comfortable saying "no" to be in charge of maintaining the interface. otherwise, and especially if the plugin devs have access to the core code, they will say stuff like "hey, I see you have a very convenient function in the core, can you expose it for my plugin?". once you open the door to this, your encapsulation suffers death by a thousand cuts. you can end up with a de facto monolithic codebase that also has a complicated plugin interface that doesn't really encapsulate anything.

[The_Colonel]

> you need to have someone who is very comfortable saying "no" to be in charge of maintaining the interface.

Of which the end result will be that the desired functionality will be somehow hacked within the plugin or will not be available at all.

Problem with such plugin based architecture is that it relies on a well designed interface. Person which designs the interface needs to have very good idea of how that interface will be used in the future, which is difficult / often impossible thing to do.

When business requirements change, you then have the difficult dilemma - just insist on "no", introduce a minimal hack, redesign interfaces to support the use case in a clean way (possibly big task) etc.

> your encapsulation suffers death by a thousand cuts. you can end up with a de facto monolithic codebase that also has a complicated plugin interface that doesn't really encapsulate anything.

Yes, worst outcome of all. In reality, plugin based architecture is no silver bullet. It can be very counter productive, especially when you're figuring out what you actually want to build, as you build it.

[leetcrew]

> When business requirements change, you then have the difficult dilemma - just insist on "no", introduce a minimal hack, redesign interfaces to support the use case in a clean way (possibly big task) etc.

one thing I will add is that every new feature does not have to be a plugin just because you have a plugin interface. "implement it directly in the core" is a perfectly valid fourth choice. some things just aren't suited to a plugin implementation.

[Jugurtha]

>one thing I will add is that every new feature does not have to be a plugin just because you have a plugin interface. "implement it directly in the core" is a perfectly valid fourth choice. some things just aren't suited to a plugin implementation.

Yes. Quoting my answer to your reply's parent:

""" It depends on the scope of the functionality. For example, right now, authentication and token generation are in the core, but it's okay right now because authentication spans across the whole product.

We eventually will extract it out, so we could use it as a component in another product, but for now, it's not inappropriate to leave it in the core. """

[Jugurtha]

>Of which the end result will be that the desired functionality will be somehow hacked within the plugin or will not be available at all.

It depends on the scope of the functionality. For example, right now, authentication and token generation are in the core, but it's okay right now because authentication spans across the whole product.

We eventually will extract it out, so we could use it as a component in another product, but for now, it's not inappropriate to leave it in the core.

>When business requirements change, you then have the difficult dilemma - just insist on "no", introduce a minimal hack, redesign interfaces to support the use case in a clean way (possibly big task) etc.

Some days are easier than others.

>Yes, worst outcome of all. In reality, plugin based architecture is no silver bullet. It can be very counter productive, especially when you're figuring out what you actually want to build, as you build it.

That's why I talked with the scope and abstraction level. We tend to make the few and loose assumptions that get us a lot of leg work done automatically. We won't make further assumptions just for 1% advantage. And if we do, there's a fallback. For example, we say that a plugin has a certain structure and expects say an icon file. If it's not there, it's not there, the plugin is loaded but just not displayed. We issue a warning, in case it was by mistake, but the application does not break.

Very few and loose "specs" that one can go through quickly and easily without looking at a checklist or something.

Again, it's not a panacea. The underlying assumption in what I wrote is that neither I nor the reader believe in silver bullets. It's not a dichotomy. The question of course is not whether a plugin architecture solves all problems and makes bacon or solves nothing, and I may have been unclear in my message. My point was that it's one of the most useful things we have done because it reduced the amount of work we had to do. We still wake up and build product.

[Jugurtha]

The core pretty much does nothing, except load the plugins and a few functions that we will extract into their components. This is what we did for the other parts.

One reason we did this was because we built custom, turn-key, ML products for large enterprise. Complete applications, from data acquisition and model training to "the JavaScript", admin interface, user management, etc.

Now... these large enterprise clients were in a sector. We could hardly sell the product to other similar clients because we couldn't just pick and choose which component or features to put on a skeleton.

It took us a lot of time, because these projects were both "software engineering" and "machine learning". In other words, we were toast. The worst of both worlds, as we were doing complete applications that even allowed their people to train models themselves.

It took a toll on morale. At some point working on eight different projects with different code bases and subsets of the team. We were fed up with this. We wanted to do things differently. We wanted to be able to get the time it took to ship the project the closest possible to the time it took to train models, which we historically did rapidly. It was all the rest that took time.

Total time = time to define problem + time to get data + time to produce models + time to write application + a big hairy epsilon

We wanted to bring "Total time" to its irreducible form. We didn't want to keep writing different applications for clients. We knew how to do it, but we did it enough times for several clients to notice patterns we wanted to extract into components. We also were losing time with the ML project lifecycle (experiment tracking, model management, collaboration, etc). We didn't want to ask the question "Which model is deployed again? What data produced that model? I tried your notebook on my machine, it doesn't work! DS: Hey, Jugurtha... Can you deploy my model? Jugurtha: I'm busy right now. I'll do it as soon as possible".

So we started building our ML platform[0] to remove as much overhead as possible, while being flexible. For example, one of our design goals is that everything one can do on the web app, they should be able to do with an API call.

- [0]: https://iko.ai

[chompychop]

Could you point me to any open source projects/references you've used for your platform that showcases this plugin architecture? I'm a junior developer and I'd really like to learn and incorporate this in my projects.

[Jugurtha]

Sure, here you go. That'll get you started. We didn't adopt any of that or a specific solution/library/way, but they were good resources to think about the matter.

That's from our internal wiki, and "Plugin Architecture" was the first entry in that, just to tell you about how important it was.

Also, given that you describe yourself as a "junior developer", here's a reply to an Ask HN about "How to become a senior developer".

https://news.ycombinator.com/item?id=25025253

Plugin Architecture:

Dynamic Code Patterns: Extending Your Application with Plugins - Doug Hellmann

Link: https://www.youtube.com/watch?v=7K72DPDOhWo

Description: Doug Hellmann talks about Stevedore, a library that provides classes for implementing common patterns for using dynamically loaded extensions

PluginBase - Armin Ronacher:

Link: http://pluginbase.pocoo.org/

Description: Armin Ronacher is the creator of Flask, Jinja, and a bunch of other stuff.

Miscellaneous links on plugin systems:

https://developer.wordpress.org/plugins

https://techsini.com/5-best-wordpress-plugin-frameworks/

https://eli.thegreenplace.net/2012/08/07/fundamental-concept...

https://pyvideo.org/pycon-us-2013/dynamic-code-patterns-exte...

https://en.wikipedia.org/wiki/Hooking

http://chateau-logic.com/content/designing-plugin-architectu...

https://www.odoo.com/documentation/user/11.0/odoo_sh/getting...

[chompychop]

Thanks a lot for taking the time to list these resources (in this post as well as in the other Ask HN post)! This is really helpful for beginners like me. :)

[echlebek]

This is an implementation of the "ports & adapters" or "hexagonal architecture" pattern. Seems like you may have independently discovered it. :)

https://en.wikipedia.org/wiki/Hexagonal_architecture_(softwa...

[Jugurtha]

I was trained as an EE so control theory, feedback systems, adapters, ports, connectors, buffers and impedance matching, transfer functions all stuck with my way of approaching designing systems, including human systems/organizations.

The way I describe it as "receptors" as in neurotransmitters is because this fascinates me. Both nicotine and acetylcholine bind to nicotinic acetylcholine receptor (nAChRs). I found that to be amazing.

[chongli]

It’s also just a specific case of the more general idea of writing libraries and common APIs instead of monolithic applications. This can be seen in the Unix philosophy with standard IO and pipelines.

[CraigJPerry]

This has been my experience too - not in ML (i have zero experience there) but in software architecture more generally. The idea of locality of change, of encapsulation, of interfaces as a way to reduce the amount of stuff you need to know in order to change the system. It is a pretty effective strategy for keeping the "cost to change" a software product low.

There have been downfalls for me though - interestingly never the obvious problem. To me the obvious problem is performance, there's usually an overhead in a plugin system of some sort but it hasn't been an issue for me yet, maybe i'm just lucky.

An annoying real issue has been adding in dependencies between plugins, that has always introduced horrible issues later on. I think based on my current life experiences i'd even choose to duplicate functionality over introducing dependencies now - despite the fact that idea is basically anethema to conventional wisdom.

[Jugurtha]

>The idea of locality of change, of encapsulation, of interfaces as a way to reduce the amount of stuff you need to know in order to change the system. It is a pretty effective strategy for keeping the "cost to change" a software product low.

Yes, not only keeping the cost of change low, but unlocking potential in a way that seems prescient but is just common sense. I like to think about this in terms of "unit of thought", "protocols, interfaces, specs", "impedance matching" to maximize power transfer, unknown future consumers.

Doing that adaptation upstream prevents people writing adapters downstream. It may be adding a "REST" API so that consumers you may ignore can work with that. It may be using Cloudevents spec for your events to make it easier for people to work with that. It may be making sure the output of your system is a Docker image to make it easier to use that elsewhere for a user, whether that may be a human or something else.

Systems that produce known/standardized/understood building blocks unlock a lot of potential and avoid downstream work.