[Animats]

Note the use case - someone wants to have the ability to replace a base-level crate such as serde.

When something near the bottom needs work, should there be a process for fixing it, which is a people problem? Or should there be a mechanism for bypassing it, which is a technical solution to a people problem? This is one of the curses of open source. The first approach means that there will be confrontations which must be resolved. The second means a proliferation of very similar packages.

This is part of the life cycle of an open source language. Early on, you don't have enough packages to get anything done, and are grateful that someone took the time to code something. Then it becomes clear that the early packages lacked something, and additional packages appear. Over time, you're drowning in cruft. In a previous posting, I mentioned ten years of getting a single standard ISO 8601 date parser adopted, instead of six packages with different bugs. Someone else went through the same exercise with Javascript.

Go tends to take the first approach, while Python takes the second. One of Go's strengths is that most of the core packages are maintained and used internally by Google. So you know they've been well-exercised.

Between Github and AI, it's all too easy to create minor variants of packages. Plus we now have package supply chain attacks. Curation has thus become more important. At this point in history, it's probably good to push towards the first approach.

[JoshTriplett]

It's a social problem that's created by a technical problem.

In many languages, if you want to integrate package A with package B, you can make and share a package AB, which people can reuse. That scales, and facilitates reuse, and avoids either package having to support everything.

In Rust, if the integration involves traits, integration between package A and package B must happen either in A or in B. That creates a scaling problem, and a social problem.

[simonask]

Other than duck-typed languages (and I count Go as basically that), which languages actually provide this feature?

AFAIK, it’s not really very common to be able to extend foreign types with new interfaces, especially not if you own neither.

C++ can technically do it using partial specialization, but it’s not exactly nice, and results in UB via ODR violation when it goes wrong (say you have two implementations of a `std::hash` specialization, etc.). And it only works for interfaces that are specifically designed to be specialized this way - not for vanilla dynamic dispatch, say.

[lmm]

> Other than duck-typed languages (and I count Go as basically that), which languages actually provide this feature?

There are only like 3 significant languages with trait-based generics, and both the other ones have some way of providing orphan instances (Haskell by requiring a flag, Scala by not having a coherence requirement at all and relying on you getting it right, which turns out to work out pretty well in practice).

More generally it's an extremely common problem to have in a mature language; if you don't have a proper fix for it then you tend to end up with awful hacks instead. Consider e.g. https://www.joda.org/joda-time-hibernate/ and https://github.com/FasterXML/jackson-datatype-joda , and note how they have to be essentially first party modules, and they have to use reflection-based runtime registries with all the associated problems. And I think that these issues significantly increased the pressure to import joda-time into the JVM system library, which ultimately came with significant downsides and costs, and in a "systems" language that aims to have a lean runtime this would be even worse.

[simonask]

Sure, the `chrono` library in Rust had essentially the same problem.

Scala is interesting. How do they resolve conflicts?

[lmm]

> Scala is interesting. How do they resolve conflicts?

If there are multiple possible instances you get a compilation error and have to specify one explicitly (which is always an option). So you do have the problem of upgrading a dependency and getting a compilation error for something that was previously fine, but it's not a big deal in practice - what I generally do is go back to the previous version and explicitly pass the instance that I was using, which is just an IDE key-combo, and then the upgrade will succeed. (After all, it's always possible to get a conflict because a library you use added a new method and the name conflicted with another library you were using - the way I see it this is essentially the same thing, just with the name being anonymous and the type being the part that matters)

You also theoretically have the much bigger problem of using two different hashing/sorting/etc. implementations with the same datastructure, which would be disastrous (although not an immediate memory corruption issue the way it could be in Rust). But in practice it's just not something I see happening, it would take a very contrived set of circumstances to encounter it.

[simonask]

Interesting, thanks for explaining.

> (although not an immediate memory corruption issue the way it could be in Rust)

Just to note, all of Rust's standard container types are designed such that they guarantee that buggy implementations of traits like `Hash` and `Ord` do not result in UB - just broken collections. :-)

[ThunderSizzle]

C# isnt a duck type language (well, you can do that via dynamic keyword, but I don't know who would do that typically).

Most integration libraries in Nuget (aka c#'s cargo) are AB type libraries.

E.g. DI Container: Autofac Messaging Library: MediatR Integration: MediatR.Extensions.Autofac.DependencyInjection

There are many examples of popular libraries like this in that world.

[simonask]

C# does not support adding interfaces to foreign types. It does support extension classes to add methods and properties to a type, but nothing that adds fields or changes the list of interfaces implemented by a type. Rust supports this as well, because you can use traits this way.

Dependency injection is a popular solution for this problem, and you can do that as well in Rust. It requires (again) that the API is designed for dependency injection, and instead of interfaces and is-a relationships, you now have "factories" producing the implementation.

[tekacs]

This is interesting but I wonder if you would accept that this also has the downside of moving at the speed of humans.

In a situation where you're building, I find the orphan rule frustrating because you can be stuck in a situation where you are unable to help yourself without forking half of the crates in the ecosystem.

Looking for improvements upstream, even with the absolute best solutions for option 1, has the fundamental downside that you can't unstick yourself.

[tekacs]

This is also where I find it surprising that this article doesn't mention Scala at all. There are MANY UX/DX challenges with the implicit and witness system in Scala, so I would never guess suggest it directly, but never have I felt more enabled to solve my own problems in a language (and yes the absolute most complex, Haskell-in-Scala libraries can absolutely an impediment to this).

With AI this pace difference is even more noticeable.

I do think that the way that Scala approaches this by using imports historically was quite interesting. Using a use statement to bring a trait definition into scope isn't discussed in any of these proposals I think?

[tadfisher]

The problem is existentials, or rather the existence of existentials without the ability to explicitly override them. Even in Haskell, overriding typeclass instances requires turning off orphan checks, which is a rather large hammer.

So once you've identified this, now you might consider the universe of possible solutions to the problem. One of those solutions might be removing existentials from your language; think about how Scala would work if implicits were removed (I haven't used Scala 3, maybe this happened?). Another solution might be to decouple the whole concept of "existential implementations of typed extension points" from libraries (or crates, or however you compile and distribute code), and require bringing instances into scope via imports or similar.

Two things are true for sure, though: libraries already depend on the current behavior, whether that makes sense or not; and forcing users to understand coherence (which instance is used by which code) is almost always a giant impediment to getting users to like your language. Hence, "orphan rules", and why everyone hates Scala 2 implicits.

[tekacs]

Yep, familiar with all of this.

That said, I would love to see a solution in my favorite class of solution: where library authors can use and benefit from this, but the average user doesn't have to notice.

I tend to think that the non-existential Scala system was _so close_, and that if you _slightly_ tweaked the scoping rules around it, you could have something great.

For example, if - as a user - I could use `.serialize(...)` from some library and it used _their_ scoped traits by default, but if I _explicitly_ (named) imported some trait(s) on my side, I could substitute my own, that'd work great.

You'd likely want to pair it with some way of e.g. allowing a per-crate prelude of explicit imports that you can ::* import within the crate to override many things at once, but... I think that with the right tweaks, you could say 'this library uses serde by default, but I can provide my own Serializer trait instead... and perhaps, if I turn off the serde Cargo feature, even their default scoped trait disappears'.

[kelnos]

That was my first thought! I never had this problem with Scala (2.x for me, but I guess there's similar syntax/concepts in 3).

The article author does talk about naming trait impls and how to use them at call sites, but never seems to consider the idea that you could import a trait impl and use it everywhere within that scope, without extra onerous syntax.

Does this still solve the "HashMap" problem though? I guess it depends on when the named impl "binds". E.g. the named Hash impl would have to bind to the HashMap itself at creation, not at calls to `insert()` or `get()`. Which... seems like a reasonable thing?

[kelnos]

> When something near the bottom needs work, should there be a process for fixing it, which is a people problem? Or should there be a mechanism for bypassing it, which is a technical solution to a people problem?

I don't think it's a people problem in the way we usually talk about the folly of creating technical solutions to people problems.

If something like serde is foundational, you simply can't radically change it without causing problems for lots and lots of people. That's a technical problem, not a people problem, even if serde needs radical change in order to evolve in the ways it needs to.

But sure, ok, let's imagine that wasn't the case. Let's say some new group of people decide that serde is lacking in some serious way, and they want to implement their changes. They can even do so without breaking compatibility with existing users of the crate. But the serde maintainers don't see the same problems; in fact, they believe that what this new group wants to do will actively cause more problems.

Neither group of people even needs to be right or wrong. Maybe both ways have pluses and minuses, and choosing just depends on what trade offs you value more. Neither group is wrong about wanting to either keep the status quo or make changes.

This is actually a technical problem: we need to find a way to allow both approaches coexist, without causing a ton of work for everyone else.

And even if we do run into situations where things need fixing, and things not getting fixed is a people problem, I'd argue for this particular sort of thing it's not only appropriate but essential that we have technical solutions to bypass the people problems. I mean, c'mon. People are people. People are going to be stubborn and not want change. Ossification is a real thing, and I think it's a rare project/organization that's able to avoid it. Sure, we could refuse to use technical workarounds when it's people we need to change, but in so many cases, that's just running up against a brick wall, over and over. Why do that to ourselves? Life is too short.

Having said that, I totally agree that there are situations where technical workarounds to people problems can be incredibly counter-productive, and cause more problems than they solve (like, "instead of expecting people to actually parent their kids, force everyone to give up their privacy for mandatory age verification; think of the children!"). But I don't think this is one of them.