[akkartik]

This tends to be mainstream opinion today: figure out the timeless essence at different scales, decompose it into abstractions, freeze their interfaces so that people can start relying on them, and so on. But it doesn't seem to have helped for forty years of trying to do it. The world changes too quickly, we aren't quite as good at designing libraries as we think, the world is filled with historical accidents in interfaces. With hindsight, it seems clear they were prematurely frozen.

Rather than dismiss these observations as isolated cases of people not practicing good behaviors, I tend to see them as evidence that we should be creating libraries far more conservatively, freezing interfaces far more late in the life cycle, perhaps even decades late.

If I'm right, we are also overusing industrial notions of assembly lines and division of labor. Libraries with non-trivial functionality take a long time to get right, and in the meantime they are produced more like guilds of craftsmen than factories. (Even if the products themselves permit factory-like operation at scale.) In that initial bake-in period we are ill-served by conventional metaphors of software components, building blocks, etc. We should be dealing more in vertically-integrated self-contained systems rather than plug-and-play libraries. More OpenBSD, less `gem install`.

I've been thinking about this for at least five years, ever since http://akkartik.name/post/libraries. More: http://akkartik.name/prose

[lmm]

Libraries do take a while to get right, but I think we can only get them right by making them as libraries. All good libraries I've seen started out as part of a vertically-integrated piece of software, but if there's currently an immature library for x, using that immature library generally puts us further along the path to getting a mature library for x than writing your own vertically-integrated implementation of x.

I'd focus on making it easier to migrate between libraries, easier to improve interfaces, and so on. (I'd argue to a certain extent that's already happened, and that's part of why we're using more and smaller libraries).

[akkartik]

Getting more users for x certainly helps mature it. However, making it a library also tends to freeze it. So ideally we'd have ways to encourage people to use something without guaranteeing its interface. I think this is a social problem; we need more libraries that have signs on them saying, "alpha software, compatibility not guaranteed," and we need greater awareness among developers that this is a good thing, that being willing to switch interfaces every once in a while results in a better eco-system in the long term.

[lmm]

Agreed. But I think the idea of listing and cutting down your dependencies (which is what I understood your post to be suggesting) is contrary to that; rather depending on a huge number of tiny libraries makes it easier for libraries to evolve more flexibly.

[akkartik]

In general I don't see the connection between the size or number of dependencies and how easy they are to evolve. Truly tiny libraries like left-pad have trivial implementations. Why not just inline them into your project? Then they're no longer libraries, just functions. You get the abstraction benefits, but you don't need to enter into a counter-party relationship with the author.

At scales above the absolutely trivial I think libraries evolve more flexibly based on social rather than technical considerations. It's about not pissing people off when you change the interface. Even something as complex as Go was able to make incompatible changes pre-1.0.

[lmm]

My sense is that smaller libraries are more able to evolve at their own place - e.g. the separation of django-rest from django core, or languages moving more things out of their standard library. Pre-1.0 is the easy part, we'll see how Go does is 5 or 10 years once it has a base of programs to maintain compatibility with - I predict it will find it harder to evolve and its library will fall behind.

[billysielu]

I agree. It's especially pungent in JavaScript, in which the average 'JavaScript developer' is playing some perverse game of library pokemon.

[graphitezepp]

Definitely stealing Javascript Pokemon. Seriously why are people still adopting more libraries to perpetuate this insanity.

[_xkjr]

Further thought: this is a similar problem to creating too many standards. https://xkcd.com/927/